2024-02-12T16:15:38.752 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-12T16:15:38.752 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --archive /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688 --owner scheduled_yuriw@teuthology --name yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688/orig.config.yaml 2024-02-12T16:15:38.800 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-12T16:15:38.883 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688 branch: wip-yuri10-testing-2024-02-08-0854-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7556688' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-02-08-0854-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-tracker-64343-pacific sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 owner: scheduled_yuriw@teuthology priority: 60 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 964 sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-tracker-64343-pacific suite_path: /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 targets: smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBrJUGIEQIwpx46m3RT+Ns3h9SYOxQsiDQZ1EmpPCSVBVxHR+WyyvvHsZqyhn1LTg9m4vWwBRcd7Pl1fB8IiJus= smithi149.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkcYX3D8sRb4OJbwChslFMKsi0ay2UAfVox0OqzpNCtXFzO0bxiVpTPp9/w9nTE8nrRKnDCJLVzcHKqmfb6ZU8= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-12_15:26:58 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-12T16:15:38.884 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa; will attempt to use it 2024-02-12T16:15:38.884 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks 2024-02-12T16:15:38.884 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-12T16:15:38.886 INFO:teuthology.task.internal:Checking packages... 2024-02-12T16:15:38.909 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0e714d9a4bd2a821113e6318adb87bd06cf81ec1' 2024-02-12T16:15:38.909 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-12T16:15:38.909 INFO:teuthology.packaging:ref: None 2024-02-12T16:15:38.910 INFO:teuthology.packaging:tag: None 2024-02-12T16:15:38.910 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-12T16:15:38.910 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:15:38.910 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-12T16:15:39.149 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-521.g0e714d9a 2024-02-12T16:15:39.151 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-12T16:15:39.158 INFO:teuthology.task.internal:no buildpackages task found 2024-02-12T16:15:39.159 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-12T16:15:39.167 INFO:teuthology.task.internal:Saving configuration 2024-02-12T16:15:39.185 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-12T16:15:39.193 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-12T16:15:39.220 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 16:09:41.952892', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBrJUGIEQIwpx46m3RT+Ns3h9SYOxQsiDQZ1EmpPCSVBVxHR+WyyvvHsZqyhn1LTg9m4vWwBRcd7Pl1fB8IiJus='} 2024-02-12T16:15:39.243 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 16:09:41.951384', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkcYX3D8sRb4OJbwChslFMKsi0ay2UAfVox0OqzpNCtXFzO0bxiVpTPp9/w9nTE8nrRKnDCJLVzcHKqmfb6ZU8='} 2024-02-12T16:15:39.243 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-12T16:15:39.250 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-12T16:15:39.250 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-12T16:15:39.250 INFO:teuthology.run_tasks:Running task console_log... 2024-02-12T16:15:39.320 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f79f513af70>, signals=[15]) 2024-02-12T16:15:39.320 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-12T16:15:39.326 INFO:teuthology.task.internal:Opening connections... 2024-02-12T16:15:39.327 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-02-12T16:15:39.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:15:39.402 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi149.front.sepia.ceph.com 2024-02-12T16:15:39.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:15:39.472 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-12T16:15:39.480 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-02-12T16:15:39.502 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-02-12T16:15:39.502 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-02-12T16:15:39.563 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-02-12T16:15:39.563 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="8" 2024-02-12T16:15:39.563 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-02-12T16:15:39.563 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-02-12T16:15:39.563 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="8" 2024-02-12T16:15:39.563 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el8" 2024-02-12T16:15:39.563 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-12T16:15:39.563 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-02-12T16:15:39.563 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-12T16:15:39.563 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-02-12T16:15:39.563 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-12T16:15:39.564 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-12T16:15:39.564 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-12T16:15:39.564 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-02-12T16:15:39.587 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2024-02-12T16:15:39.609 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2024-02-12T16:15:39.609 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2024-02-12T16:15:39.671 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="CentOS Stream" 2024-02-12T16:15:39.671 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8" 2024-02-12T16:15:39.671 INFO:teuthology.orchestra.run.smithi149.stdout:ID="centos" 2024-02-12T16:15:39.671 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="rhel fedora" 2024-02-12T16:15:39.671 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8" 2024-02-12T16:15:39.671 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8" 2024-02-12T16:15:39.672 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-12T16:15:39.672 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2024-02-12T16:15:39.672 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-12T16:15:39.672 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://centos.org/" 2024-02-12T16:15:39.672 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-12T16:15:39.672 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-12T16:15:39.672 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-12T16:15:39.672 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2024-02-12T16:15:39.694 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-12T16:15:39.703 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-12T16:15:39.710 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-12T16:15:39.711 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-02-12T16:15:39.713 DEBUG:teuthology.orchestra.run.smithi149:> test '!' -e /home/ubuntu/cephtest 2024-02-12T16:15:39.733 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-12T16:15:39.740 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-12T16:15:39.740 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-02-12T16:15:39.774 DEBUG:teuthology.orchestra.run.smithi149:> test -z $(ls -A /var/lib/ceph) 2024-02-12T16:15:39.801 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-12T16:15:39.855 INFO:teuthology.run_tasks:Running task kernel... 2024-02-12T16:15:39.870 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-12T16:15:39.870 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-12T16:15:39.870 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-12T16:15:39.871 DEBUG:teuthology.orchestra.run.smithi149:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-12T16:15:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:39.890 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-02-12T16:15:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:39.891 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-02-12T16:15:39.948 INFO:teuthology.orchestra.run.smithi149.stdout:4.18.0-489.el8.x86_64 2024-02-12T16:15:39.949 INFO:teuthology.task.kernel:Running kernel on smithi149: 4.18.0-489.el8.x86_64 2024-02-12T16:15:39.949 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum install -y kernel 2024-02-12T16:15:39.952 INFO:teuthology.orchestra.run.smithi104.stdout:4.18.0-489.el8.x86_64 2024-02-12T16:15:39.952 INFO:teuthology.task.kernel:Running kernel on smithi104: 4.18.0-489.el8.x86_64 2024-02-12T16:15:39.952 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2024-02-12T16:15:48.401 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 8 - AppStream 4.6 MB/s | 27 MB 00:05 2024-02-12T16:15:48.927 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - AppStream 4.2 MB/s | 27 MB 00:06 2024-02-12T16:15:54.142 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 10 MB 00:01 2024-02-12T16:15:54.535 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-02-12T16:15:56.336 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-02-12T16:15:56.540 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - Extras 46 kB/s | 18 kB 00:00 2024-02-12T16:15:56.760 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.9 kB 00:00 2024-02-12T16:15:56.963 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.9 kB 00:00 2024-02-12T16:15:56.987 INFO:teuthology.orchestra.run.smithi104.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-02-12T16:15:57.214 INFO:teuthology.orchestra.run.smithi149.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-02-12T16:15:57.987 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2024-02-12T16:15:58.155 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2024-02-12T16:16:01.712 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 437 kB/s | 24 kB 00:00 2024-02-12T16:16:02.022 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 316 kB/s | 24 kB 00:00 2024-02-12T16:16:04.625 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-12T16:16:04.625 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-12T16:16:04.732 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:16:04.733 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:16:04.733 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-12T16:16:04.733 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:16:04.733 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:16:04.733 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-12T16:16:04.733 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-12T16:16:04.733 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-12T16:16:04.733 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-12T16:16:04.734 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:16:04.734 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:16:04.734 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:16:04.734 INFO:teuthology.orchestra.run.smithi104.stdout:Install 3 Packages 2024-02-12T16:16:04.734 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:16:04.734 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 90 M 2024-02-12T16:16:04.734 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 96 M 2024-02-12T16:16:04.735 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:16:04.976 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-12T16:16:04.976 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-12T16:16:05.077 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:16:05.077 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:16:05.077 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-02-12T16:16:05.078 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:16:05.078 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:16:05.078 INFO:teuthology.orchestra.run.smithi149.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-12T16:16:05.078 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-02-12T16:16:05.078 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-12T16:16:05.078 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-12T16:16:05.078 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:16:05.078 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:16:05.078 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:16:05.078 INFO:teuthology.orchestra.run.smithi149.stdout:Install 3 Packages 2024-02-12T16:16:05.078 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:16:05.079 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 90 M 2024-02-12T16:16:05.079 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 96 M 2024-02-12T16:16:05.079 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:16:06.137 INFO:teuthology.orchestra.run.smithi104.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 8.5 MB/s | 10 MB 00:01 2024-02-12T16:16:06.615 INFO:teuthology.orchestra.run.smithi149.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 6.9 MB/s | 10 MB 00:01 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi104.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 12 MB/s | 36 MB 00:03 2024-02-12T16:16:07.981 INFO:teuthology.orchestra.run.smithi149.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 13 MB/s | 36 MB 00:02 2024-02-12T16:16:08.355 INFO:teuthology.orchestra.run.smithi104.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 13 MB/s | 44 MB 00:03 2024-02-12T16:16:08.355 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:16:08.355 INFO:teuthology.orchestra.run.smithi104.stdout:Total 25 MB/s | 90 MB 00:03 2024-02-12T16:16:08.507 INFO:teuthology.orchestra.run.smithi149.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 13 MB/s | 44 MB 00:03 2024-02-12T16:16:08.507 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:16:08.507 INFO:teuthology.orchestra.run.smithi149.stdout:Total 26 MB/s | 90 MB 00:03 2024-02-12T16:16:09.034 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:16:09.196 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:16:09.474 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:16:09.475 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:16:09.634 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:16:09.634 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:16:10.718 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:16:10.751 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:16:10.883 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:16:10.905 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:16:13.554 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:16:13.664 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:16:14.018 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:16:14.151 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:16:17.585 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:16:17.724 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:16:17.992 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:16:18.109 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:16:21.421 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:16:21.555 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:16:21.698 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:16:21.885 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:16:43.612 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:16:43.636 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:16:43.756 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:16:43.779 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:16:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:16:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:16:44.024 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:16:44.142 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:16:44.142 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-12T16:16:44.142 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-12T16:16:44.468 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:16:44.468 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:16:44.468 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:16:44.468 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-12T16:16:44.468 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-12T16:16:44.468 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:16:44.468 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:16:44.636 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-02-12T16:16:44.666 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-12T16:16:44.666 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:16:44.666 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:16:44.666 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-12T16:16:44.666 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-12T16:16:44.666 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:16:44.666 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:16:44.801 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-02-12T16:16:44.851 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-12T16:16:44.852 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-12T16:16:44.852 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-12T16:16:44.852 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-12T16:16:44.852 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-12T16:16:44.852 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-12T16:16:44.852 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-02-12T16:16:44.853 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-02-12T16:16:45.025 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-12T16:16:45.026 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-12T16:16:45.026 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-12T16:16:45.026 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-12T16:16:45.026 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-12T16:16:45.026 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-12T16:16:45.026 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi149.front.sepia.ceph.com, path=None, version=distro) 2024-02-12T16:16:45.026 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-02-12T16:16:45.063 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-12T16:16:45.063 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-12T16:16:45.063 DEBUG:teuthology.orchestra.run.smithi104:> sudo rpm -qi grub2-tools 2024-02-12T16:16:45.096 INFO:teuthology.orchestra.run.smithi104.stdout:Name : grub2-tools 2024-02-12T16:16:45.096 INFO:teuthology.orchestra.run.smithi104.stdout:Epoch : 1 2024-02-12T16:16:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Version : 2.02 2024-02-12T16:16:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Release : 129.el8 2024-02-12T16:16:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: x86_64 2024-02-12T16:16:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-12T16:16:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Group : System Environment/Base 2024-02-12T16:16:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Size : 9265586 2024-02-12T16:16:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:License : GPLv3+ 2024-02-12T16:16:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-12T16:16:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-12T16:16:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-12T16:16:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout:Relocations : (not relocatable) 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout:Packager : CentOS BuildSystem 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout:Vendor : CentOS 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout:Summary : Support tools for GRUB. 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout:Description : 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout:hardware devices. 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:16:45.098 INFO:teuthology.orchestra.run.smithi104.stdout:This subpackage provides tools for support of all platforms. 2024-02-12T16:16:45.099 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-12T16:16:45.100 INFO:teuthology.task.kernel:Updating grub on smithi104 to boot 4.18.0-536.el8.x86_64 2024-02-12T16:16:45.100 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-12T16:16:45.242 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-12T16:16:45.242 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-12T16:16:45.242 DEBUG:teuthology.orchestra.run.smithi149:> sudo rpm -qi grub2-tools 2024-02-12T16:16:45.276 INFO:teuthology.orchestra.run.smithi149.stdout:Name : grub2-tools 2024-02-12T16:16:45.276 INFO:teuthology.orchestra.run.smithi149.stdout:Epoch : 1 2024-02-12T16:16:45.276 INFO:teuthology.orchestra.run.smithi149.stdout:Version : 2.02 2024-02-12T16:16:45.276 INFO:teuthology.orchestra.run.smithi149.stdout:Release : 129.el8 2024-02-12T16:16:45.276 INFO:teuthology.orchestra.run.smithi149.stdout:Architecture: x86_64 2024-02-12T16:16:45.276 INFO:teuthology.orchestra.run.smithi149.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-12T16:16:45.276 INFO:teuthology.orchestra.run.smithi149.stdout:Group : System Environment/Base 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:Size : 9265586 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:License : GPLv3+ 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:Relocations : (not relocatable) 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:Packager : CentOS BuildSystem 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:Vendor : CentOS 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:Summary : Support tools for GRUB. 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:Description : 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:16:45.277 INFO:teuthology.orchestra.run.smithi149.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-12T16:16:45.278 INFO:teuthology.orchestra.run.smithi149.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-12T16:16:45.278 INFO:teuthology.orchestra.run.smithi149.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-12T16:16:45.278 INFO:teuthology.orchestra.run.smithi149.stdout:hardware devices. 2024-02-12T16:16:45.278 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:16:45.278 INFO:teuthology.orchestra.run.smithi149.stdout:This subpackage provides tools for support of all platforms. 2024-02-12T16:16:45.279 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-12T16:16:45.279 INFO:teuthology.task.kernel:Updating grub on smithi149 to boot 4.18.0-536.el8.x86_64 2024-02-12T16:16:45.279 DEBUG:teuthology.orchestra.run.smithi149:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-12T16:16:46.753 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-02-12T16:16:46.796 INFO:teuthology.orchestra.run.smithi149.stderr:Generating grub configuration file ... 2024-02-12T16:16:48.891 INFO:teuthology.orchestra.run.smithi149.stderr:done 2024-02-12T16:16:48.893 DEBUG:teuthology.orchestra.run.smithi149:> mktemp 2024-02-12T16:16:48.907 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-02-12T16:16:48.909 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-02-12T16:16:48.928 INFO:teuthology.orchestra.run.smithi149.stdout:/tmp/tmp.paACxPivro 2024-02-12T16:16:48.929 DEBUG:teuthology.orchestra.run.smithi149:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.paACxPivro 2024-02-12T16:16:48.945 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.AE8JS66TQT 2024-02-12T16:16:48.945 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AE8JS66TQT 2024-02-12T16:16:49.002 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0666 /tmp/tmp.paACxPivro 2024-02-12T16:16:49.018 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.AE8JS66TQT 2024-02-12T16:16:49.148 DEBUG:teuthology.orchestra.remote:smithi149:/tmp/tmp.paACxPivro is 6KB 2024-02-12T16:16:49.167 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.AE8JS66TQT is 6KB 2024-02-12T16:16:49.196 DEBUG:teuthology.orchestra.run.smithi149:> rm -fr /tmp/tmp.paACxPivro 2024-02-12T16:16:49.213 DEBUG:teuthology.orchestra.run.smithi149:> sudo /bin/ls /boot/loader/entries || true 2024-02-12T16:16:49.216 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.AE8JS66TQT 2024-02-12T16:16:49.234 DEBUG:teuthology.orchestra.run.smithi104:> sudo /bin/ls /boot/loader/entries || true 2024-02-12T16:16:49.285 INFO:teuthology.orchestra.run.smithi149.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-12T16:16:49.285 INFO:teuthology.orchestra.run.smithi149.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-12T16:16:49.285 INFO:teuthology.orchestra.run.smithi149.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-12T16:16:49.285 INFO:teuthology.orchestra.run.smithi149.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-12T16:16:49.286 DEBUG:teuthology.orchestra.run.smithi149:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-12T16:16:49.313 INFO:teuthology.orchestra.run.smithi104.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-12T16:16:49.313 INFO:teuthology.orchestra.run.smithi104.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-12T16:16:49.313 INFO:teuthology.orchestra.run.smithi104.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-12T16:16:49.313 INFO:teuthology.orchestra.run.smithi104.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-12T16:16:49.315 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-12T16:16:49.433 DEBUG:teuthology.orchestra.run.smithi149:> sudo shutdown -r now 2024-02-12T16:16:49.493 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-02-12T16:17:19.464 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-12T16:17:19.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-12T16:17:19.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:17:19.496 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-12T16:17:19.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-12T16:17:19.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:17:37.856 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-02-12T16:17:37.857 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-12T16:17:46.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-12T16:17:46.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:17:46.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-12T16:17:46.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:17:49.916 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-02-12T16:17:49.948 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-12T16:18:01.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-12T16:18:01.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:18:01.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-12T16:18:01.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:18:20.348 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.149 2024-02-12T16:18:34.426 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-02-12T16:18:35.224 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-12T16:18:35.224 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-536.el8.x86_64"... 2024-02-12T16:18:35.224 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-02-12T16:18:35.241 INFO:teuthology.orchestra.run.smithi104.stdout:4.18.0-536.el8.x86_64 2024-02-12T16:18:35.242 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-12T16:18:35.242 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-12T16:18:35.243 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-12T16:18:35.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-12T16:18:35.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:18:35.644 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-02-12T16:18:35.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-12T16:18:35.970 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-536.el8.x86_64"... 2024-02-12T16:18:35.970 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-02-12T16:18:35.987 INFO:teuthology.orchestra.run.smithi149.stdout:4.18.0-536.el8.x86_64 2024-02-12T16:18:35.987 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-12T16:18:35.987 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-12T16:18:35.987 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-12T16:18:36.244 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-12T16:18:36.323 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-12T16:18:36.398 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-02-12T16:18:36.411 DEBUG:teuthology.parallel:result is None 2024-02-12T16:18:36.988 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-12T16:18:36.988 DEBUG:teuthology.orchestra.run.smithi149:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-12T16:18:37.733 INFO:teuthology.orchestra.run.smithi149.stdout:ttyS1 2024-02-12T16:18:37.764 DEBUG:teuthology.parallel:result is None 2024-02-12T16:18:37.765 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-12T16:18:37.773 INFO:teuthology.task.internal:Creating test directory... 2024-02-12T16:18:37.774 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-12T16:18:37.776 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-12T16:18:37.950 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-12T16:18:37.961 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-12T16:18:37.976 INFO:teuthology.task.internal:Creating archive directory... 2024-02-12T16:18:37.976 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-12T16:18:37.978 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-12T16:18:38.269 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-12T16:18:38.277 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-12T16:18:38.278 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-12T16:18:38.281 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-12T16:18:38.313 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T16:18:38.319 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T16:18:38.323 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T16:18:38.329 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T16:18:38.331 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-12T16:18:38.342 INFO:teuthology.task.internal:Configuring sudo... 2024-02-12T16:18:38.342 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-12T16:18:38.373 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-12T16:18:38.403 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-12T16:18:38.416 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-12T16:18:38.416 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-12T16:18:38.445 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-12T16:18:38.463 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-12T16:18:38.550 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-12T16:18:38.607 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-12T16:18:38.694 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-12T16:18:38.735 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-12T16:18:38.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-12T16:18:38.803 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-12T16:18:38.852 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-12T16:18:38.895 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-12T16:18:38.978 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-12T16:18:39.015 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-12T16:18:39.015 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-12T16:18:39.083 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-02-12T16:18:39.085 DEBUG:teuthology.orchestra.run.smithi149:> sudo service rsyslog restart 2024-02-12T16:18:39.138 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-12T16:18:39.202 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-12T16:18:39.622 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-12T16:18:39.632 INFO:teuthology.task.internal:Starting timer... 2024-02-12T16:18:39.632 INFO:teuthology.run_tasks:Running task pcp... 2024-02-12T16:18:39.674 INFO:teuthology.run_tasks:Running task selinux... 2024-02-12T16:18:39.688 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-12T16:18:39.688 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-02-12T16:18:39.781 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-02-12T16:18:39.783 DEBUG:teuthology.orchestra.run.smithi149:> sudo service auditd rotate 2024-02-12T16:18:39.892 INFO:teuthology.orchestra.run.smithi149.stdout:Rotating logs: 2024-02-12T16:18:39.894 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-12T16:18:39.895 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-02-12T16:18:39.915 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-02-12T16:18:39.915 DEBUG:teuthology.orchestra.run.smithi149:> /usr/sbin/getenforce 2024-02-12T16:18:39.954 INFO:teuthology.orchestra.run.smithi149.stdout:Permissive 2024-02-12T16:18:39.954 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi104.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi149.front.sepia.ceph.com': 'permissive'} 2024-02-12T16:18:39.954 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-12T16:18:39.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:39.982 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-12T16:18:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:40.021 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-12T16:18:40.021 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-02-12T16:18:40.058 DEBUG:teuthology.orchestra.run.smithi149:> sudo /usr/sbin/setenforce permissive 2024-02-12T16:18:40.086 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-12T16:18:40.099 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-12T16:18:40.104 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-12T16:18:40.215 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-12T16:18:40.232 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-12T16:18:40.233 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi104.front.sepia.ceph.com,smithi149.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-12T16:24:26.828 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi149.front.sepia.ceph.com')] 2024-02-12T16:24:26.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-12T16:24:26.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:24:26.905 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-02-12T16:24:26.978 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-12T16:24:26.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-12T16:24:26.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T16:24:27.053 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-02-12T16:24:27.123 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-02-12T16:24:27.124 INFO:teuthology.run_tasks:Running task clock... 2024-02-12T16:24:27.135 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-12T16:24:27.135 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-12T16:24:27.136 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T16:24:27.138 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-12T16:24:27.139 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T16:24:27.170 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-12T16:24:27.185 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-12T16:24:27.197 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-12T16:24:27.211 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-12T16:24:27.251 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-02-12T16:24:27.262 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-02-12T16:24:27.276 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-12T16:24:27.288 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: ntpd: command not found 2024-02-12T16:24:27.290 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-12T16:24:27.299 INFO:teuthology.orchestra.run.smithi149.stdout:506 Cannot talk to daemon 2024-02-12T16:24:27.313 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-12T16:24:27.327 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-12T16:24:27.356 INFO:teuthology.orchestra.run.smithi104.stderr:bash: ntpq: command not found 2024-02-12T16:24:27.358 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-12T16:24:27.359 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-02-12T16:24:27.359 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:24:27.359 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:24:27.359 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:24:27.359 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:24:27.399 INFO:teuthology.orchestra.run.smithi149.stderr:bash: ntpq: command not found 2024-02-12T16:24:27.402 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-12T16:24:27.402 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-02-12T16:24:27.402 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:24:27.402 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:24:27.402 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:24:27.402 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:24:27.403 INFO:teuthology.run_tasks:Running task pexec... 2024-02-12T16:24:27.412 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-12T16:24:27.413 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi104.front.sepia.ceph.com 2024-02-12T16:24:27.413 DEBUG:teuthology.orchestra.run.smithi104:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-12T16:24:27.413 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi149.front.sepia.ceph.com 2024-02-12T16:24:27.413 DEBUG:teuthology.orchestra.run.smithi149:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-12T16:24:28.193 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:22 ago on Mon 12 Feb 2024 04:23:06 PM UTC. 2024-02-12T16:24:28.245 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:22 ago on Mon 12 Feb 2024 04:23:06 PM UTC. 2024-02-12T16:24:30.168 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:24:30.169 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:24:30.169 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-12T16:24:30.169 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:24:30.169 INFO:teuthology.orchestra.run.smithi104.stdout:Resetting modules: 2024-02-12T16:24:30.170 INFO:teuthology.orchestra.run.smithi104.stdout: container-tools 2024-02-12T16:24:30.170 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:24:30.170 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:24:30.170 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:24:30.170 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:24:30.190 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:24:30.191 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:24:30.191 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-02-12T16:24:30.191 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:24:30.191 INFO:teuthology.orchestra.run.smithi149.stdout:Resetting modules: 2024-02-12T16:24:30.191 INFO:teuthology.orchestra.run.smithi149.stdout: container-tools 2024-02-12T16:24:30.191 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:24:30.191 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:24:30.191 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:24:30.192 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:24:30.457 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:24:30.460 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:24:31.330 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:25 ago on Mon 12 Feb 2024 04:23:06 PM UTC. 2024-02-12T16:24:31.334 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:25 ago on Mon 12 Feb 2024 04:23:06 PM UTC. 2024-02-12T16:24:33.311 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:24:33.315 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:24:33.346 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:24:33.346 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-02-12T16:24:33.346 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:24:33.346 INFO:teuthology.orchestra.run.smithi149.stdout:Installing group/module packages: 2024-02-12T16:24:33.346 INFO:teuthology.orchestra.run.smithi149.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-12T16:24:33.346 INFO:teuthology.orchestra.run.smithi149.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-12T16:24:33.346 INFO:teuthology.orchestra.run.smithi149.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-12T16:24:33.346 INFO:teuthology.orchestra.run.smithi149.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-12T16:24:33.346 INFO:teuthology.orchestra.run.smithi149.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout:Installing weak dependencies: 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout:Installing module profiles: 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout: container-tools/common 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:24:33.347 INFO:teuthology.orchestra.run.smithi149.stdout:Enabling module streams: 2024-02-12T16:24:33.348 INFO:teuthology.orchestra.run.smithi149.stdout: container-tools rhel8 2024-02-12T16:24:33.348 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:24:33.348 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:24:33.348 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:24:33.348 INFO:teuthology.orchestra.run.smithi149.stdout:Install 13 Packages 2024-02-12T16:24:33.348 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:24:33.349 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 27 M 2024-02-12T16:24:33.349 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 91 M 2024-02-12T16:24:33.349 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:24:33.351 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:24:33.351 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-12T16:24:33.351 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:24:33.351 INFO:teuthology.orchestra.run.smithi104.stdout:Installing group/module packages: 2024-02-12T16:24:33.351 INFO:teuthology.orchestra.run.smithi104.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-12T16:24:33.351 INFO:teuthology.orchestra.run.smithi104.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-12T16:24:33.351 INFO:teuthology.orchestra.run.smithi104.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-12T16:24:33.351 INFO:teuthology.orchestra.run.smithi104.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-12T16:24:33.351 INFO:teuthology.orchestra.run.smithi104.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-12T16:24:33.351 INFO:teuthology.orchestra.run.smithi104.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout:Installing weak dependencies: 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout:Installing module profiles: 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout: container-tools/common 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:24:33.352 INFO:teuthology.orchestra.run.smithi104.stdout:Enabling module streams: 2024-02-12T16:24:33.353 INFO:teuthology.orchestra.run.smithi104.stdout: container-tools rhel8 2024-02-12T16:24:33.353 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:24:33.353 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:24:33.353 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:24:33.353 INFO:teuthology.orchestra.run.smithi104.stdout:Install 13 Packages 2024-02-12T16:24:33.353 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:24:33.353 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 27 M 2024-02-12T16:24:33.354 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 91 M 2024-02-12T16:24:33.354 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:24:33.926 INFO:teuthology.orchestra.run.smithi104.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 1.5 MB/s | 759 kB 00:00 2024-02-12T16:24:34.010 INFO:teuthology.orchestra.run.smithi104.stdout:(2/13): aardvark-dns-1.9.0-1.module_el8+804+f13 1.8 MB/s | 1.0 MB 00:00 2024-02-12T16:24:34.027 INFO:teuthology.orchestra.run.smithi104.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 378 kB/s | 38 kB 00:00 2024-02-12T16:24:34.077 INFO:teuthology.orchestra.run.smithi149.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 1.6 MB/s | 759 kB 00:00 2024-02-12T16:24:34.135 INFO:teuthology.orchestra.run.smithi104.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 1.9 MB/s | 248 kB 00:00 2024-02-12T16:24:34.152 INFO:teuthology.orchestra.run.smithi149.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 505 kB/s | 38 kB 00:00 2024-02-12T16:24:34.186 INFO:teuthology.orchestra.run.smithi149.stdout:(3/13): aardvark-dns-1.9.0-1.module_el8+804+f13 1.7 MB/s | 1.0 MB 00:00 2024-02-12T16:24:34.277 INFO:teuthology.orchestra.run.smithi104.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 1.2 MB/s | 179 kB 00:00 2024-02-12T16:24:34.344 INFO:teuthology.orchestra.run.smithi104.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 372 kB/s | 25 kB 00:00 2024-02-12T16:24:34.411 INFO:teuthology.orchestra.run.smithi104.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2024-02-12T16:24:34.469 INFO:teuthology.orchestra.run.smithi149.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 784 kB/s | 248 kB 00:00 2024-02-12T16:24:34.595 INFO:teuthology.orchestra.run.smithi149.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 1.4 MB/s | 179 kB 00:00 2024-02-12T16:24:34.795 INFO:teuthology.orchestra.run.smithi149.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 124 kB/s | 25 kB 00:00 2024-02-12T16:24:35.062 INFO:teuthology.orchestra.run.smithi104.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 3.8 MB/s | 4.0 MB 00:01 2024-02-12T16:24:35.120 INFO:teuthology.orchestra.run.smithi149.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 289 kB/s | 94 kB 00:00 2024-02-12T16:24:35.279 INFO:teuthology.orchestra.run.smithi149.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 3.6 MB/s | 4.0 MB 00:01 2024-02-12T16:24:35.745 INFO:teuthology.orchestra.run.smithi104.stdout:(9/13): toolbox-0.0.99.4-8.module_el8+804+f1313 3.7 MB/s | 2.5 MB 00:00 2024-02-12T16:24:35.804 INFO:teuthology.orchestra.run.smithi104.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 902 kB/s | 53 kB 00:00 2024-02-12T16:24:35.954 INFO:teuthology.orchestra.run.smithi149.stdout:(9/13): toolbox-0.0.99.4-8.module_el8+804+f1313 3.7 MB/s | 2.5 MB 00:00 2024-02-12T16:24:35.978 INFO:teuthology.orchestra.run.smithi104.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 235 kB/s | 41 kB 00:00 2024-02-12T16:24:35.996 INFO:teuthology.orchestra.run.smithi149.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 1.2 MB/s | 53 kB 00:00 2024-02-12T16:24:36.055 INFO:teuthology.orchestra.run.smithi149.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 704 kB/s | 41 kB 00:00 2024-02-12T16:24:36.571 INFO:teuthology.orchestra.run.smithi104.stdout:(12/13): buildah-1.34.0-1.module_el8+804+f13139 3.1 MB/s | 9.7 MB 00:03 2024-02-12T16:24:36.805 INFO:teuthology.orchestra.run.smithi104.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 3.7 MB/s | 8.8 MB 00:02 2024-02-12T16:24:36.805 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:24:36.805 INFO:teuthology.orchestra.run.smithi104.stdout:Total 7.9 MB/s | 27 MB 00:03 2024-02-12T16:24:36.888 INFO:teuthology.orchestra.run.smithi149.stdout:(12/13): buildah-1.34.0-1.module_el8+804+f13139 2.9 MB/s | 9.7 MB 00:03 2024-02-12T16:24:37.047 INFO:teuthology.orchestra.run.smithi149.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 4.6 MB/s | 8.8 MB 00:01 2024-02-12T16:24:37.047 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:24:37.047 INFO:teuthology.orchestra.run.smithi149.stdout:Total 7.4 MB/s | 27 MB 00:03 2024-02-12T16:24:37.076 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:24:37.097 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:24:37.097 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:24:37.322 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:24:37.343 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:24:37.343 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:24:37.371 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:24:37.374 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:24:37.599 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:24:37.602 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:24:38.088 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:24:38.263 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:24:38.304 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-12T16:24:38.499 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-12T16:24:38.503 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-12T16:24:38.673 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-12T16:24:38.684 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-12T16:24:38.859 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-12T16:24:38.934 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-12T16:24:39.075 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-12T16:24:39.481 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-12T16:24:39.614 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-12T16:24:40.436 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-12T16:24:40.582 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-12T16:24:40.691 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-12T16:24:40.843 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-12T16:24:40.970 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-12T16:24:41.047 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-12T16:24:41.253 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-12T16:24:41.302 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-12T16:24:41.669 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-12T16:24:41.696 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-12T16:24:42.611 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-12T16:24:42.661 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-12T16:24:42.828 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-12T16:24:42.853 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-12T16:24:43.031 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-12T16:24:43.080 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-12T16:24:43.459 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-12T16:24:43.459 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-12T16:24:43.459 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-12T16:24:43.459 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-12T16:24:43.459 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-12T16:24:43.460 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-12T16:24:43.460 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-12T16:24:43.460 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-12T16:24:43.460 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-12T16:24:43.460 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-12T16:24:43.460 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-12T16:24:43.460 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-12T16:24:43.460 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-12T16:24:43.522 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-12T16:24:43.522 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-12T16:24:43.522 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-12T16:24:43.522 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-12T16:24:43.522 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-12T16:24:43.523 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-12T16:24:43.523 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-12T16:24:43.523 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-12T16:24:43.523 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-12T16:24:43.523 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-12T16:24:43.523 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-12T16:24:43.523 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-12T16:24:43.523 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-12T16:24:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-12T16:24:43.935 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-12T16:24:43.936 INFO:teuthology.orchestra.run.smithi104.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-12T16:24:43.937 INFO:teuthology.orchestra.run.smithi104.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-12T16:24:43.937 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:24:43.937 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:24:44.019 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-12T16:24:44.019 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:24:44.019 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:24:44.019 INFO:teuthology.orchestra.run.smithi149.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-12T16:24:44.020 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:24:44.021 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:24:44.280 DEBUG:teuthology.parallel:result is None 2024-02-12T16:24:44.285 DEBUG:teuthology.parallel:result is None 2024-02-12T16:24:44.285 INFO:teuthology.run_tasks:Running task pexec... 2024-02-12T16:24:44.294 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-12T16:24:44.294 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi104.front.sepia.ceph.com 2024-02-12T16:24:44.295 DEBUG:teuthology.orchestra.run.smithi104:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-12T16:24:44.295 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi149.front.sepia.ceph.com 2024-02-12T16:24:44.295 DEBUG:teuthology.orchestra.run.smithi149:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-12T16:24:44.475 DEBUG:teuthology.parallel:result is None 2024-02-12T16:24:44.500 DEBUG:teuthology.parallel:result is None 2024-02-12T16:24:44.501 INFO:teuthology.run_tasks:Running task install... 2024-02-12T16:24:44.510 DEBUG:teuthology.task.install:project ceph 2024-02-12T16:24:44.511 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}} 2024-02-12T16:24:44.511 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-12T16:24:44.511 INFO:teuthology.task.install:Using flavor: default 2024-02-12T16:24:44.519 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-12T16:24:44.519 INFO:teuthology.task.install:extra packages: [] 2024-02-12T16:24:44.519 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-12T16:24:44.519 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-12T16:24:44.519 INFO:teuthology.packaging:ref: None 2024-02-12T16:24:44.519 INFO:teuthology.packaging:tag: v16.2.4 2024-02-12T16:24:44.519 INFO:teuthology.packaging:branch: None 2024-02-12T16:24:44.520 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:24:44.746 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-12T16:24:44.746 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-12T16:24:44.749 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-12T16:24:44.749 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-12T16:24:44.750 INFO:teuthology.packaging:ref: None 2024-02-12T16:24:44.750 INFO:teuthology.packaging:tag: v16.2.4 2024-02-12T16:24:44.750 INFO:teuthology.packaging:branch: None 2024-02-12T16:24:44.750 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:24:44.750 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-12T16:24:44.893 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-12T16:24:44.893 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-12T16:24:44.926 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-12T16:24:44.926 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-12T16:24:44.926 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-12T16:24:44.957 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-12T16:24:44.957 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-12T16:24:44.957 INFO:teuthology.packaging:ref: None 2024-02-12T16:24:44.957 INFO:teuthology.packaging:tag: v16.2.4 2024-02-12T16:24:44.957 INFO:teuthology.packaging:branch: None 2024-02-12T16:24:44.957 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:24:44.958 DEBUG:teuthology.orchestra.run.smithi149:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-12T16:24:44.979 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-12T16:24:44.979 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-12T16:24:45.005 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-12T16:24:45.005 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-12T16:24:45.006 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-12T16:24:45.028 DEBUG:teuthology.orchestra.run.smithi149:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-12T16:24:45.033 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-12T16:24:45.034 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-12T16:24:45.034 INFO:teuthology.packaging:ref: None 2024-02-12T16:24:45.034 INFO:teuthology.packaging:tag: v16.2.4 2024-02-12T16:24:45.034 INFO:teuthology.packaging:branch: None 2024-02-12T16:24:45.034 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:24:45.034 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-12T16:24:45.102 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-12T16:24:45.105 DEBUG:teuthology.orchestra.run.smithi149:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-12T16:24:45.171 INFO:teuthology.orchestra.run.smithi149.stdout:check_obsoletes = 1 2024-02-12T16:24:45.173 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum clean all 2024-02-12T16:24:45.178 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-12T16:24:45.244 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2024-02-12T16:24:45.245 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2024-02-12T16:24:45.447 INFO:teuthology.orchestra.run.smithi149.stdout:76 files removed 2024-02-12T16:24:45.477 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-radosgw 2024-02-12T16:24:45.506 INFO:teuthology.orchestra.run.smithi104.stdout:76 files removed 2024-02-12T16:24:45.533 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw 2024-02-12T16:24:45.972 INFO:teuthology.orchestra.run.smithi149.stdout:ceph packages for x86_64 1.0 MB/s | 74 kB 00:00 2024-02-12T16:24:46.081 INFO:teuthology.orchestra.run.smithi149.stdout:ceph noarch packages 214 kB/s | 15 kB 00:00 2024-02-12T16:24:46.138 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 373 kB/s | 74 kB 00:00 2024-02-12T16:24:46.165 INFO:teuthology.orchestra.run.smithi149.stdout:ceph source packages 32 kB/s | 1.7 kB 00:00 2024-02-12T16:24:46.272 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 159 kB/s | 15 kB 00:00 2024-02-12T16:24:46.381 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-02-12T16:24:52.379 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 8 - AppStream 4.5 MB/s | 27 MB 00:05 2024-02-12T16:24:52.424 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - AppStream 4.3 MB/s | 27 MB 00:06 2024-02-12T16:24:58.023 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-02-12T16:24:58.191 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-02-12T16:25:00.156 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 8 - Extras 34 kB/s | 18 kB 00:00 2024-02-12T16:25:00.166 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2024-02-12T16:25:00.565 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 6.9 kB 00:00 2024-02-12T16:25:00.592 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.9 kB 00:00 2024-02-12T16:25:01.276 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 8 - PowerTools 8.0 MB/s | 4.8 MB 00:00 2024-02-12T16:25:01.292 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 8 - PowerTools 7.5 MB/s | 4.8 MB 00:00 2024-02-12T16:25:02.403 INFO:teuthology.orchestra.run.smithi149.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-02-12T16:25:02.409 INFO:teuthology.orchestra.run.smithi104.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-02-12T16:25:03.719 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-02-12T16:25:04.831 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 6.9 MB/s | 16 MB 00:02 2024-02-12T16:25:07.560 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 254 kB/s | 24 kB 00:00 2024-02-12T16:25:08.612 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 569 kB/s | 24 kB 00:00 2024-02-12T16:25:11.108 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:25:11.110 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-12T16:25:11.111 INFO:teuthology.orchestra.run.smithi104.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-12T16:25:11.112 INFO:teuthology.orchestra.run.smithi104.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout:Install 25 Packages 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 2 Packages 2024-02-12T16:25:11.113 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:25:11.114 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 59 M 2024-02-12T16:25:11.114 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:25:12.102 INFO:teuthology.orchestra.run.smithi104.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 45 MB/s | 6.0 MB 00:00 2024-02-12T16:25:12.119 INFO:teuthology.orchestra.run.smithi104.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2024-02-12T16:25:12.155 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:25:12.158 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:25:12.158 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-02-12T16:25:12.158 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:25:12.158 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:25:12.158 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout:Upgrading: 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-12T16:25:12.159 INFO:teuthology.orchestra.run.smithi149.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-12T16:25:12.160 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-12T16:25:12.161 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-12T16:25:12.161 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:25:12.161 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:25:12.161 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:25:12.161 INFO:teuthology.orchestra.run.smithi149.stdout:Install 25 Packages 2024-02-12T16:25:12.161 INFO:teuthology.orchestra.run.smithi149.stdout:Upgrade 2 Packages 2024-02-12T16:25:12.161 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:25:12.162 INFO:teuthology.orchestra.run.smithi104.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-02-12T16:25:12.162 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 59 M 2024-02-12T16:25:12.162 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:25:12.195 INFO:teuthology.orchestra.run.smithi104.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-12T16:25:12.287 INFO:teuthology.orchestra.run.smithi104.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2024-02-12T16:25:12.303 INFO:teuthology.orchestra.run.smithi104.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2024-02-12T16:25:12.329 INFO:teuthology.orchestra.run.smithi104.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-12T16:25:12.387 INFO:teuthology.orchestra.run.smithi104.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 213 kB 00:00 2024-02-12T16:25:12.421 INFO:teuthology.orchestra.run.smithi104.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-12T16:25:12.455 INFO:teuthology.orchestra.run.smithi104.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-02-12T16:25:12.480 INFO:teuthology.orchestra.run.smithi104.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-12T16:25:12.818 INFO:teuthology.orchestra.run.smithi104.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 15 MB/s | 13 MB 00:00 2024-02-12T16:25:13.201 INFO:teuthology.orchestra.run.smithi104.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 202 kB/s | 145 kB 00:00 2024-02-12T16:25:13.239 INFO:teuthology.orchestra.run.smithi149.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 12 MB/s | 6.0 MB 00:00 2024-02-12T16:25:13.281 INFO:teuthology.orchestra.run.smithi149.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 602 kB/s | 25 kB 00:00 2024-02-12T16:25:13.314 INFO:teuthology.orchestra.run.smithi149.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-02-12T16:25:13.356 INFO:teuthology.orchestra.run.smithi149.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2024-02-12T16:25:13.448 INFO:teuthology.orchestra.run.smithi149.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2024-02-12T16:25:13.473 INFO:teuthology.orchestra.run.smithi149.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-12T16:25:13.499 INFO:teuthology.orchestra.run.smithi149.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-12T16:25:13.524 INFO:teuthology.orchestra.run.smithi149.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-02-12T16:25:13.566 INFO:teuthology.orchestra.run.smithi149.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 9.1 MB/s | 387 kB 00:00 2024-02-12T16:25:13.599 INFO:teuthology.orchestra.run.smithi149.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-02-12T16:25:13.625 INFO:teuthology.orchestra.run.smithi149.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-12T16:25:13.758 INFO:teuthology.orchestra.run.smithi149.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 13 MB/s | 13 MB 00:01 2024-02-12T16:25:13.774 INFO:teuthology.orchestra.run.smithi104.stdout:(14/27): ceph-common-16.2.4-0.el8.x86_64.rpm 13 MB/s | 24 MB 00:01 2024-02-12T16:25:13.799 INFO:teuthology.orchestra.run.smithi104.stdout:(15/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 434 kB/s | 259 kB 00:00 2024-02-12T16:25:13.841 INFO:teuthology.orchestra.run.smithi104.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 662 kB/s | 677 kB 00:01 2024-02-12T16:25:13.867 INFO:teuthology.orchestra.run.smithi104.stdout:(17/27): python3-prettytable-0.7.2-14.el8.noarc 477 kB/s | 44 kB 00:00 2024-02-12T16:25:14.042 INFO:teuthology.orchestra.run.smithi149.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 19 MB/s | 24 MB 00:01 2024-02-12T16:25:14.050 INFO:teuthology.orchestra.run.smithi104.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 168 kB/s | 42 kB 00:00 2024-02-12T16:25:14.067 INFO:teuthology.orchestra.run.smithi149.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 328 kB/s | 145 kB 00:00 2024-02-12T16:25:14.075 INFO:teuthology.orchestra.run.smithi104.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 201 kB/s | 47 kB 00:00 2024-02-12T16:25:14.118 INFO:teuthology.orchestra.run.smithi149.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 879 kB/s | 44 kB 00:00 2024-02-12T16:25:14.143 INFO:teuthology.orchestra.run.smithi104.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 291 kB/s | 80 kB 00:00 2024-02-12T16:25:14.143 INFO:teuthology.orchestra.run.smithi149.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.7 MB/s | 677 kB 00:00 2024-02-12T16:25:14.176 INFO:teuthology.orchestra.run.smithi149.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.9 MB/s | 259 kB 00:00 2024-02-12T16:25:14.385 INFO:teuthology.orchestra.run.smithi149.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 158 kB/s | 42 kB 00:00 2024-02-12T16:25:14.410 INFO:teuthology.orchestra.run.smithi149.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 176 kB/s | 47 kB 00:00 2024-02-12T16:25:14.434 INFO:teuthology.orchestra.run.smithi104.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 303 kB/s | 116 kB 00:00 2024-02-12T16:25:14.444 INFO:teuthology.orchestra.run.smithi149.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 300 kB/s | 80 kB 00:00 2024-02-12T16:25:14.476 INFO:teuthology.orchestra.run.smithi104.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 765 kB/s | 306 kB 00:00 2024-02-12T16:25:14.518 INFO:teuthology.orchestra.run.smithi104.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 494 kB/s | 181 kB 00:00 2024-02-12T16:25:14.544 INFO:teuthology.orchestra.run.smithi104.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 540 kB/s | 59 kB 00:00 2024-02-12T16:25:14.561 INFO:teuthology.orchestra.run.smithi104.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 896 kB/s | 75 kB 00:00 2024-02-12T16:25:14.644 INFO:teuthology.orchestra.run.smithi104.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2024-02-12T16:25:14.669 INFO:teuthology.orchestra.run.smithi149.stdout:(21/27): leveldb-1.22-1.el8.x86_64.rpm 804 kB/s | 181 kB 00:00 2024-02-12T16:25:14.703 INFO:teuthology.orchestra.run.smithi149.stdout:(22/27): fmt-6.2.1-1.el8.x86_64.rpm 366 kB/s | 116 kB 00:00 2024-02-12T16:25:14.728 INFO:teuthology.orchestra.run.smithi149.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2024-02-12T16:25:14.736 INFO:teuthology.orchestra.run.smithi104.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 21 MB/s | 4.0 MB 00:00 2024-02-12T16:25:14.740 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:25:14.741 INFO:teuthology.orchestra.run.smithi104.stdout:Total 16 MB/s | 59 MB 00:03 2024-02-12T16:25:14.762 INFO:teuthology.orchestra.run.smithi149.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 872 kB/s | 306 kB 00:00 2024-02-12T16:25:14.795 INFO:teuthology.orchestra.run.smithi149.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 813 kB/s | 75 kB 00:00 2024-02-12T16:25:14.810 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:25:14.858 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:25:14.858 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:25:14.862 INFO:teuthology.orchestra.run.smithi149.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-02-12T16:25:14.921 INFO:teuthology.orchestra.run.smithi149.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 25 MB/s | 4.0 MB 00:00 2024-02-12T16:25:14.923 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:25:14.923 INFO:teuthology.orchestra.run.smithi149.stdout:Total 21 MB/s | 59 MB 00:02 2024-02-12T16:25:14.995 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:25:15.044 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:25:15.045 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:25:15.204 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:25:15.206 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:25:15.383 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:25:15.386 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:25:16.353 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:25:16.515 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:25:16.587 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-12T16:25:16.739 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-12T16:25:16.817 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-12T16:25:16.927 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-12T16:25:16.975 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-12T16:25:17.076 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-12T16:25:17.369 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-12T16:25:17.472 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-12T16:25:17.583 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-12T16:25:17.651 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-12T16:25:17.678 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-12T16:25:17.729 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-12T16:25:17.946 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-12T16:25:17.973 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-12T16:25:18.083 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-12T16:25:18.126 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-12T16:25:18.181 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-12T16:25:18.191 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-12T16:25:18.376 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-12T16:25:18.393 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-12T16:25:18.911 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-12T16:25:18.916 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-12T16:25:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-12T16:25:19.084 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-12T16:25:19.156 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-12T16:25:19.165 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-12T16:25:19.339 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-12T16:25:19.379 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-12T16:25:19.500 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-12T16:25:19.549 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-12T16:25:19.557 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-12T16:25:19.629 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-12T16:25:19.714 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-12T16:25:19.813 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-12T16:25:19.884 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-12T16:25:19.999 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-12T16:25:20.042 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-12T16:25:20.100 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-12T16:25:20.166 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-12T16:25:20.229 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-12T16:25:20.287 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-12T16:25:20.407 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-12T16:25:20.720 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-12T16:25:20.841 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-12T16:25:20.900 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-12T16:25:20.999 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-12T16:25:21.049 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-12T16:25:21.145 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-12T16:25:21.179 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-12T16:25:21.345 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-12T16:25:21.410 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-12T16:25:21.542 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-12T16:25:21.566 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-12T16:25:21.619 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-12T16:25:21.707 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-12T16:25:21.789 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-12T16:25:21.810 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-12T16:25:21.983 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-12T16:25:22.000 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-12T16:25:22.166 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-12T16:25:22.232 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-12T16:25:22.383 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-12T16:25:22.439 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-12T16:25:22.632 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-12T16:25:24.939 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-12T16:25:25.167 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-12T16:25:25.202 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-12T16:25:25.424 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-12T16:25:25.870 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-12T16:25:26.130 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-12T16:25:26.149 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-12T16:25:26.260 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-12T16:25:26.391 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-12T16:25:26.444 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-12T16:25:26.580 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-12T16:25:26.731 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-12T16:25:45.826 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-12T16:25:45.826 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2024-02-12T16:25:45.826 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2024-02-12T16:25:45.826 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /dev 2024-02-12T16:25:45.826 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /run 2024-02-12T16:25:45.826 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2024-02-12T16:25:45.826 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2024-02-12T16:25:45.826 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2024-02-12T16:25:45.826 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2024-02-12T16:25:45.826 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /dev 2024-02-12T16:25:45.826 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:25:45.972 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-12T16:25:45.972 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /sys 2024-02-12T16:25:45.973 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /proc 2024-02-12T16:25:45.973 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /dev 2024-02-12T16:25:45.973 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /run 2024-02-12T16:25:45.973 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /mnt 2024-02-12T16:25:45.973 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /var/tmp 2024-02-12T16:25:45.973 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /home 2024-02-12T16:25:45.973 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /tmp 2024-02-12T16:25:45.973 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /dev 2024-02-12T16:25:45.973 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:25:47.148 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-12T16:25:47.260 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-12T16:25:47.260 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-12T16:25:47.279 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-12T16:25:47.351 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-12T16:25:47.351 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-12T16:25:47.445 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-12T16:25:47.446 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-12T16:25:47.565 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-12T16:25:47.565 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-12T16:25:48.127 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-12T16:25:48.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-12T16:25:48.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-12T16:25:48.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-12T16:25:48.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-12T16:25:48.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-12T16:25:48.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-12T16:25:48.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-12T16:25:48.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-12T16:25:48.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-12T16:25:48.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-12T16:25:48.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-12T16:25:48.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-12T16:25:48.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-12T16:25:48.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-12T16:25:48.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-12T16:25:48.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-12T16:25:48.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-12T16:25:48.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-12T16:25:48.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-12T16:25:48.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-12T16:25:48.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-12T16:25:48.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-12T16:25:48.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-12T16:25:48.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-12T16:25:48.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-12T16:25:48.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-12T16:25:48.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-12T16:25:48.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-12T16:25:48.245 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-12T16:25:48.246 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-12T16:25:48.246 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-12T16:25:48.246 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-12T16:25:48.246 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-12T16:25:48.246 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-12T16:25:48.246 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-12T16:25:48.246 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-12T16:25:48.246 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-12T16:25:48.246 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-12T16:25:48.246 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-12T16:25:48.246 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-12T16:25:48.247 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-12T16:25:48.248 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-12T16:25:48.248 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-12T16:25:48.248 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-12T16:25:48.248 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-12T16:25:48.248 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-12T16:25:49.042 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-12T16:25:49.042 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:25:49.042 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout: leveldb-1.22-1.el8.x86_64 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.043 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-12T16:25:49.044 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.045 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.045 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.045 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:25:49.045 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:25:49.186 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-12T16:25:49.186 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:25:49.187 INFO:teuthology.orchestra.run.smithi149.stdout:Upgraded: 2024-02-12T16:25:49.187 INFO:teuthology.orchestra.run.smithi149.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.187 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:25:49.187 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.187 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.187 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.187 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.187 INFO:teuthology.orchestra.run.smithi149.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-12T16:25:49.187 INFO:teuthology.orchestra.run.smithi149.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-12T16:25:49.187 INFO:teuthology.orchestra.run.smithi149.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: leveldb-1.22-1.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.188 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.189 INFO:teuthology.orchestra.run.smithi149.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-12T16:25:49.189 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.189 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.189 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-12T16:25:49.189 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:25:49.189 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:25:49.374 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-test 2024-02-12T16:25:49.434 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-test 2024-02-12T16:25:50.127 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:43 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:25:50.210 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:42 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:25:51.639 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:25:51.640 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:25:51.640 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-12T16:25:51.640 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:25:51.640 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:25:51.640 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-12T16:25:51.640 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-12T16:25:51.640 INFO:teuthology.orchestra.run.smithi104.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-12T16:25:51.640 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-12T16:25:51.641 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-12T16:25:51.641 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-12T16:25:51.641 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-12T16:25:51.641 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:25:51.641 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:25:51.641 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:25:51.641 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-02-12T16:25:51.641 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:25:51.642 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 53 M 2024-02-12T16:25:51.642 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 229 M 2024-02-12T16:25:51.642 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:25:51.722 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:25:51.723 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:25:51.723 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-02-12T16:25:51.723 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:25:51.723 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:25:51.723 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-12T16:25:51.723 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-02-12T16:25:51.724 INFO:teuthology.orchestra.run.smithi149.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-12T16:25:51.724 INFO:teuthology.orchestra.run.smithi149.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-12T16:25:51.724 INFO:teuthology.orchestra.run.smithi149.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-12T16:25:51.724 INFO:teuthology.orchestra.run.smithi149.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-12T16:25:51.724 INFO:teuthology.orchestra.run.smithi149.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-12T16:25:51.724 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:25:51.724 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:25:51.724 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:25:51.724 INFO:teuthology.orchestra.run.smithi149.stdout:Install 6 Packages 2024-02-12T16:25:51.724 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:25:51.725 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 53 M 2024-02-12T16:25:51.725 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 229 M 2024-02-12T16:25:51.725 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:25:51.868 INFO:teuthology.orchestra.run.smithi149.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 169 kB 00:00 2024-02-12T16:25:51.901 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 169 kB 00:00 2024-02-12T16:25:51.909 INFO:teuthology.orchestra.run.smithi149.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.5 MB/s | 203 kB 00:00 2024-02-12T16:25:52.010 INFO:teuthology.orchestra.run.smithi149.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.2 MB/s | 323 kB 00:00 2024-02-12T16:25:52.035 INFO:teuthology.orchestra.run.smithi149.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-02-12T16:25:52.060 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 925 kB/s | 203 kB 00:00 2024-02-12T16:25:52.060 INFO:teuthology.orchestra.run.smithi149.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-02-12T16:25:52.085 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-02-12T16:25:52.135 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-02-12T16:25:52.194 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2024-02-12T16:25:52.602 INFO:teuthology.orchestra.run.smithi149.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 63 MB/s | 52 MB 00:00 2024-02-12T16:25:52.604 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:25:52.605 INFO:teuthology.orchestra.run.smithi149.stdout:Total 60 MB/s | 53 MB 00:00 2024-02-12T16:25:52.648 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:25:52.669 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:25:52.669 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:25:52.711 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 60 MB/s | 52 MB 00:00 2024-02-12T16:25:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:25:52.713 INFO:teuthology.orchestra.run.smithi104.stdout:Total 49 MB/s | 53 MB 00:01 2024-02-12T16:25:52.755 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:25:52.776 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:25:52.776 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:25:52.993 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:25:52.996 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:25:53.093 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:25:53.096 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:25:53.476 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:25:53.571 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:25:53.698 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-12T16:25:53.786 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-12T16:25:53.906 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-12T16:25:53.969 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-12T16:25:54.062 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-12T16:25:54.132 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-12T16:25:54.133 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-12T16:25:54.198 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-12T16:25:54.295 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-12T16:25:54.363 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-12T16:25:54.437 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-12T16:25:54.542 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-12T16:25:59.317 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-12T16:25:59.399 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-12T16:25:59.577 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-12T16:25:59.664 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-12T16:25:59.883 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-12T16:25:59.883 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-12T16:25:59.883 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-12T16:25:59.883 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-12T16:25:59.883 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-12T16:25:59.883 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-12T16:25:59.952 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-12T16:25:59.952 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-12T16:25:59.952 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-12T16:25:59.952 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-12T16:25:59.952 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-12T16:25:59.952 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-12T16:26:00.361 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-12T16:26:00.361 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:00.361 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:26:00.361 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-12T16:26:00.361 INFO:teuthology.orchestra.run.smithi149.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-12T16:26:00.361 INFO:teuthology.orchestra.run.smithi149.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-12T16:26:00.361 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:00.361 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:26:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-12T16:26:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:00.418 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:26:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-12T16:26:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-12T16:26:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-12T16:26:00.418 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:00.419 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:26:00.592 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph 2024-02-12T16:26:00.633 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph 2024-02-12T16:26:01.370 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:53 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:26:01.385 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:54 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:26:02.874 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:26:02.878 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:02.878 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-02-12T16:26:02.878 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:02.878 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:26:02.878 INFO:teuthology.orchestra.run.smithi149.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-12T16:26:02.878 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-02-12T16:26:02.878 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-12T16:26:02.878 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-12T16:26:02.878 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-12T16:26:02.878 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-12T16:26:02.878 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-12T16:26:02.879 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-12T16:26:02.880 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:02.881 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:26:02.881 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:02.881 INFO:teuthology.orchestra.run.smithi149.stdout:Install 32 Packages 2024-02-12T16:26:02.881 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:02.881 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 31 M 2024-02-12T16:26:02.881 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 111 M 2024-02-12T16:26:02.881 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:26:02.885 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:26:02.889 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:02.889 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-12T16:26:02.889 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:02.889 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:26:02.889 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-12T16:26:02.889 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-12T16:26:02.889 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-12T16:26:02.889 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-12T16:26:02.889 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-12T16:26:02.889 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-12T16:26:02.890 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-12T16:26:02.891 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-12T16:26:02.892 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:02.892 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:26:02.892 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:02.892 INFO:teuthology.orchestra.run.smithi104.stdout:Install 32 Packages 2024-02-12T16:26:02.892 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:02.892 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 31 M 2024-02-12T16:26:02.893 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 111 M 2024-02-12T16:26:02.893 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:26:03.261 INFO:teuthology.orchestra.run.smithi104.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 94 kB/s | 6.3 kB 00:00 2024-02-12T16:26:03.303 INFO:teuthology.orchestra.run.smithi104.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-02-12T16:26:03.362 INFO:teuthology.orchestra.run.smithi104.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-02-12T16:26:03.364 INFO:teuthology.orchestra.run.smithi149.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 134 kB/s | 6.3 kB 00:00 2024-02-12T16:26:03.404 INFO:teuthology.orchestra.run.smithi104.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 5.2 MB/s | 220 kB 00:00 2024-02-12T16:26:03.414 INFO:teuthology.orchestra.run.smithi149.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-02-12T16:26:03.456 INFO:teuthology.orchestra.run.smithi149.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 17 MB/s | 2.4 MB 00:00 2024-02-12T16:26:03.488 INFO:teuthology.orchestra.run.smithi104.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-02-12T16:26:03.489 INFO:teuthology.orchestra.run.smithi149.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-02-12T16:26:03.598 INFO:teuthology.orchestra.run.smithi149.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-02-12T16:26:03.632 INFO:teuthology.orchestra.run.smithi149.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-02-12T16:26:03.730 INFO:teuthology.orchestra.run.smithi104.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 43 MB/s | 18 MB 00:00 2024-02-12T16:26:03.755 INFO:teuthology.orchestra.run.smithi104.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 145 kB/s | 39 kB 00:00 2024-02-12T16:26:03.805 INFO:teuthology.orchestra.run.smithi104.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 391 kB/s | 157 kB 00:00 2024-02-12T16:26:03.830 INFO:teuthology.orchestra.run.smithi104.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 756 kB/s | 57 kB 00:00 2024-02-12T16:26:03.849 INFO:teuthology.orchestra.run.smithi149.stdout:(7/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 42 MB/s | 18 MB 00:00 2024-02-12T16:26:03.856 INFO:teuthology.orchestra.run.smithi104.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 821 kB/s | 103 kB 00:00 2024-02-12T16:26:03.866 INFO:teuthology.orchestra.run.smithi149.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 145 kB/s | 39 kB 00:00 2024-02-12T16:26:03.891 INFO:teuthology.orchestra.run.smithi149.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 400 kB/s | 103 kB 00:00 2024-02-12T16:26:03.933 INFO:teuthology.orchestra.run.smithi149.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 682 kB/s | 57 kB 00:00 2024-02-12T16:26:04.000 INFO:teuthology.orchestra.run.smithi149.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.3 MB/s | 457 kB 00:00 2024-02-12T16:26:04.040 INFO:teuthology.orchestra.run.smithi104.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.9 MB/s | 457 kB 00:00 2024-02-12T16:26:04.207 INFO:teuthology.orchestra.run.smithi104.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 661 kB/s | 249 kB 00:00 2024-02-12T16:26:04.251 INFO:teuthology.orchestra.run.smithi149.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 693 kB/s | 249 kB 00:00 2024-02-12T16:26:04.274 INFO:teuthology.orchestra.run.smithi104.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.5 MB/s | 641 kB 00:00 2024-02-12T16:26:04.290 INFO:teuthology.orchestra.run.smithi104.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 522 kB/s | 44 kB 00:00 2024-02-12T16:26:04.293 INFO:teuthology.orchestra.run.smithi149.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 373 kB/s | 109 kB 00:00 2024-02-12T16:26:04.332 INFO:teuthology.orchestra.run.smithi104.stdout:(15/32): python3-pycparser-2.14-14.el8.noarch.r 373 kB/s | 109 kB 00:00 2024-02-12T16:26:04.383 INFO:teuthology.orchestra.run.smithi104.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.8 MB/s | 173 kB 00:00 2024-02-12T16:26:04.401 INFO:teuthology.orchestra.run.smithi149.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.3 MB/s | 641 kB 00:00 2024-02-12T16:26:04.408 INFO:teuthology.orchestra.run.smithi104.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 445 kB/s | 11 kB 00:00 2024-02-12T16:26:04.433 INFO:teuthology.orchestra.run.smithi104.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 707 kB/s | 18 kB 00:00 2024-02-12T16:26:04.485 INFO:teuthology.orchestra.run.smithi104.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.5 MB/s | 384 kB 00:00 2024-02-12T16:26:04.509 INFO:teuthology.orchestra.run.smithi104.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 646 kB/s | 49 kB 00:00 2024-02-12T16:26:04.526 INFO:teuthology.orchestra.run.smithi104.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.4 MB/s | 59 kB 00:00 2024-02-12T16:26:04.559 INFO:teuthology.orchestra.run.smithi104.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2024-02-12T16:26:04.576 INFO:teuthology.orchestra.run.smithi104.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2024-02-12T16:26:04.585 INFO:teuthology.orchestra.run.smithi149.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 131 kB/s | 44 kB 00:00 2024-02-12T16:26:04.593 INFO:teuthology.orchestra.run.smithi104.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 558 kB/s | 19 kB 00:00 2024-02-12T16:26:04.610 INFO:teuthology.orchestra.run.smithi104.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 715 kB/s | 24 kB 00:00 2024-02-12T16:26:04.627 INFO:teuthology.orchestra.run.smithi104.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 874 kB/s | 29 kB 00:00 2024-02-12T16:26:04.644 INFO:teuthology.orchestra.run.smithi104.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 820 kB/s | 27 kB 00:00 2024-02-12T16:26:04.678 INFO:teuthology.orchestra.run.smithi104.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 5.0 MB/s | 255 kB 00:00 2024-02-12T16:26:04.705 INFO:teuthology.orchestra.run.smithi104.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.1 MB/s | 251 kB 00:00 2024-02-12T16:26:04.728 INFO:teuthology.orchestra.run.smithi104.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-02-12T16:26:04.768 INFO:teuthology.orchestra.run.smithi149.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 389 kB/s | 185 kB 00:00 2024-02-12T16:26:04.778 INFO:teuthology.orchestra.run.smithi104.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 316 kB/s | 23 kB 00:00 2024-02-12T16:26:04.862 INFO:teuthology.orchestra.run.smithi104.stdout:(32/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 315 kB/s | 185 kB 00:00 2024-02-12T16:26:04.863 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:04.864 INFO:teuthology.orchestra.run.smithi104.stdout:Total 16 MB/s | 31 MB 00:01 2024-02-12T16:26:04.894 INFO:teuthology.orchestra.run.smithi149.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2024-02-12T16:26:04.927 INFO:teuthology.orchestra.run.smithi149.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 328 kB/s | 173 kB 00:00 2024-02-12T16:26:04.943 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:26:04.982 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:26:04.982 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:26:05.003 INFO:teuthology.orchestra.run.smithi149.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 161 kB/s | 18 kB 00:00 2024-02-12T16:26:05.028 INFO:teuthology.orchestra.run.smithi149.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 868 kB/s | 384 kB 00:00 2024-02-12T16:26:05.053 INFO:teuthology.orchestra.run.smithi149.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 387 kB/s | 49 kB 00:00 2024-02-12T16:26:05.112 INFO:teuthology.orchestra.run.smithi149.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 540 kB/s | 59 kB 00:00 2024-02-12T16:26:05.154 INFO:teuthology.orchestra.run.smithi149.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2024-02-12T16:26:05.212 INFO:teuthology.orchestra.run.smithi149.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2024-02-12T16:26:05.272 INFO:teuthology.orchestra.run.smithi149.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 204 kB/s | 24 kB 00:00 2024-02-12T16:26:05.285 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:26:05.288 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:26:05.296 INFO:teuthology.orchestra.run.smithi149.stdout:(26/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.0 MB/s | 283 kB 00:00 2024-02-12T16:26:05.322 INFO:teuthology.orchestra.run.smithi149.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 268 kB/s | 29 kB 00:00 2024-02-12T16:26:05.372 INFO:teuthology.orchestra.run.smithi149.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2024-02-12T16:26:05.489 INFO:teuthology.orchestra.run.smithi149.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 733 kB/s | 85 kB 00:00 2024-02-12T16:26:05.514 INFO:teuthology.orchestra.run.smithi149.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2024-02-12T16:26:05.589 INFO:teuthology.orchestra.run.smithi149.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2024-02-12T16:26:05.631 INFO:teuthology.orchestra.run.smithi149.stdout:(32/32): python3-waitress-1.4.3-1.el8.noarch.rp 763 kB/s | 255 kB 00:00 2024-02-12T16:26:05.633 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:05.633 INFO:teuthology.orchestra.run.smithi149.stdout:Total 11 MB/s | 31 MB 00:02 2024-02-12T16:26:05.712 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:26:05.751 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:26:05.751 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:26:06.079 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:26:06.081 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:26:06.464 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:26:06.711 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-12T16:26:06.939 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-12T16:26:07.179 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-12T16:26:07.194 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:26:07.407 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-12T16:26:07.420 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-12T16:26:07.617 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-12T16:26:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-12T16:26:07.815 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-12T16:26:07.867 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-12T16:26:08.017 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-12T16:26:08.066 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-12T16:26:08.177 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-12T16:26:08.230 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-12T16:26:08.402 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-12T16:26:08.410 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-12T16:26:08.562 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-12T16:26:08.614 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-12T16:26:08.782 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-12T16:26:08.836 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-12T16:26:08.973 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-12T16:26:09.111 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-12T16:26:09.169 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-12T16:26:09.374 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-12T16:26:09.536 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-12T16:26:09.613 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-12T16:26:09.825 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-12T16:26:09.982 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-12T16:26:09.996 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-12T16:26:10.229 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-12T16:26:10.273 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-12T16:26:10.400 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-12T16:26:10.507 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-12T16:26:10.633 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-12T16:26:10.802 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-12T16:26:10.816 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-12T16:26:11.052 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-12T16:26:11.076 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-12T16:26:11.258 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-12T16:26:11.281 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-12T16:26:11.469 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-12T16:26:11.485 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-12T16:26:11.661 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-12T16:26:11.733 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-12T16:26:11.906 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-12T16:26:12.009 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-12T16:26:12.152 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-12T16:26:12.285 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-12T16:26:12.393 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-12T16:26:12.611 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-12T16:26:12.707 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-12T16:26:12.919 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-12T16:26:12.990 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-12T16:26:13.296 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-12T16:26:13.380 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-12T16:26:13.484 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-12T16:26:13.600 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-12T16:26:15.022 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-12T16:26:15.130 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-12T16:26:15.208 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-12T16:26:15.325 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-12T16:26:15.641 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-12T16:26:15.758 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-12T16:26:15.850 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-12T16:26:15.966 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-12T16:26:16.073 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-12T16:26:16.188 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-12T16:26:16.254 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-12T16:26:16.272 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-12T16:26:16.351 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-12T16:26:16.368 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-12T16:26:16.429 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-12T16:26:16.500 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-12T16:26:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-12T16:26:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-12T16:26:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-12T16:26:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-12T16:26:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-12T16:26:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-12T16:26:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-12T16:26:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-12T16:26:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-12T16:26:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-12T16:26:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-12T16:26:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-12T16:26:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-12T16:26:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-12T16:26:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-12T16:26:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-12T16:26:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-12T16:26:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-12T16:26:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-12T16:26:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-12T16:26:16.937 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-12T16:26:16.937 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-12T16:26:16.937 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-12T16:26:16.937 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-12T16:26:16.937 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-12T16:26:16.937 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-12T16:26:16.937 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-12T16:26:16.937 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-12T16:26:16.937 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-12T16:26:16.937 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-12T16:26:16.938 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-12T16:26:16.939 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-12T16:26:16.939 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-12T16:26:16.940 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-12T16:26:16.940 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-12T16:26:16.940 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-12T16:26:16.940 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-12T16:26:16.940 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-12T16:26:17.770 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-12T16:26:17.770 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:17.770 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:26:17.770 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:17.770 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:17.770 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:17.770 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-12T16:26:17.770 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:17.770 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:17.770 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-2.6-1.el8.noarch 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-12T16:26:17.771 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:17.772 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:26:17.808 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-12T16:26:17.809 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-portend-2.6-1.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-12T16:26:17.810 INFO:teuthology.orchestra.run.smithi149.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-12T16:26:17.811 INFO:teuthology.orchestra.run.smithi149.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-12T16:26:17.811 INFO:teuthology.orchestra.run.smithi149.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-12T16:26:17.811 INFO:teuthology.orchestra.run.smithi149.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-12T16:26:17.811 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-12T16:26:17.811 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-12T16:26:17.811 INFO:teuthology.orchestra.run.smithi149.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-12T16:26:17.811 INFO:teuthology.orchestra.run.smithi149.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-12T16:26:17.811 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:17.811 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:26:18.053 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-base 2024-02-12T16:26:18.068 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-base 2024-02-12T16:26:18.803 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:11 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:26:18.837 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:10 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:26:20.261 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:26:20.281 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:26:20.306 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:26:20.308 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-12T16:26:20.308 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:26:20.328 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:26:20.329 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-02-12T16:26:20.329 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:26:20.437 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install cephadm 2024-02-12T16:26:20.440 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install cephadm 2024-02-12T16:26:21.189 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:14 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:26:21.205 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:13 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:26:22.688 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:26:22.689 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-12T16:26:22.689 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:22.689 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:26:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-12T16:26:22.689 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:22.689 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:26:22.689 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:22.690 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-12T16:26:22.690 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:22.690 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 71 k 2024-02-12T16:26:22.690 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 290 k 2024-02-12T16:26:22.690 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:26:22.728 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:26:22.729 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:22.729 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-02-12T16:26:22.729 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:22.729 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:26:22.729 INFO:teuthology.orchestra.run.smithi149.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-12T16:26:22.729 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:22.730 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:26:22.730 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:22.730 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-02-12T16:26:22.730 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:22.730 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 71 k 2024-02-12T16:26:22.730 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 290 k 2024-02-12T16:26:22.730 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:26:22.741 INFO:teuthology.orchestra.run.smithi104.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2024-02-12T16:26:22.741 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:22.742 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.3 MB/s | 71 kB 00:00 2024-02-12T16:26:22.742 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:26:22.749 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:26:22.749 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:26:22.750 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:26:22.753 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:26:22.772 INFO:teuthology.orchestra.run.smithi149.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 71 kB 00:00 2024-02-12T16:26:22.773 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:22.773 INFO:teuthology.orchestra.run.smithi149.stdout:Total 1.6 MB/s | 71 kB 00:00 2024-02-12T16:26:22.773 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:26:22.780 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:26:22.780 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:26:22.782 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:26:22.785 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:26:22.800 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:26:22.832 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:26:23.260 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-12T16:26:23.348 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-12T16:26:23.445 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-12T16:26:23.511 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-12T16:26:23.677 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-12T16:26:23.755 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-12T16:26:24.068 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-12T16:26:24.068 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:24.068 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:26:24.069 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-12T16:26:24.069 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:24.069 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:26:24.126 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-12T16:26:24.126 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:24.127 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:26:24.127 INFO:teuthology.orchestra.run.smithi149.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-12T16:26:24.127 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:24.127 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:26:24.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-immutable-object-cache 2024-02-12T16:26:24.332 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-immutable-object-cache 2024-02-12T16:26:25.079 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:18 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:26:25.112 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:17 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:26:26.568 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:26:26.568 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:26.568 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-02-12T16:26:26.568 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:26.568 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:26:26.569 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-12T16:26:26.569 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:26.569 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:26:26.569 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:26.569 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-12T16:26:26.569 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:26.569 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 163 k 2024-02-12T16:26:26.569 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 587 k 2024-02-12T16:26:26.569 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:26:26.612 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:26:26.613 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:26.613 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repo Size 2024-02-12T16:26:26.613 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:26.614 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:26:26.614 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-12T16:26:26.614 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:26.614 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:26:26.614 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:26.614 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-02-12T16:26:26.614 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:26.614 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 163 k 2024-02-12T16:26:26.614 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 587 k 2024-02-12T16:26:26.614 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:26:27.073 INFO:teuthology.orchestra.run.smithi149.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 355 kB/s | 163 kB 00:00 2024-02-12T16:26:27.074 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:27.074 INFO:teuthology.orchestra.run.smithi149.stdout:Total 354 kB/s | 163 kB 00:00 2024-02-12T16:26:27.075 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:26:27.087 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:26:27.087 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:26:27.091 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 312 kB/s | 163 kB 00:00 2024-02-12T16:26:27.092 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:27.093 INFO:teuthology.orchestra.run.smithi104.stdout:Total 311 kB/s | 163 kB 00:00 2024-02-12T16:26:27.093 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:26:27.105 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:26:27.105 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:26:27.140 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:26:27.143 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:26:27.156 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:26:27.160 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:26:27.277 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:26:27.464 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:26:27.465 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:26:27.693 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:26:27.890 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:26:28.100 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:26:28.269 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:26:28.269 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:28.269 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:26:28.269 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:28.269 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:28.269 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:26:28.489 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:26:28.489 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:28.489 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:26:28.489 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-12T16:26:28.489 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:28.489 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:26:28.514 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr 2024-02-12T16:26:28.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr 2024-02-12T16:26:29.296 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:21 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:26:29.506 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:22 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:26:30.755 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:26:30.801 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:26:30.803 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-02-12T16:26:30.803 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:26:30.897 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-dashboard 2024-02-12T16:26:30.971 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:26:31.018 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:26:31.019 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-12T16:26:31.019 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:26:31.142 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-dashboard 2024-02-12T16:26:31.666 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:23 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:26:31.891 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:24 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:26:33.166 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:26:33.167 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:33.167 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-02-12T16:26:33.167 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:33.167 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:26:33.167 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-12T16:26:33.167 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-02-12T16:26:33.167 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-12T16:26:33.167 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-12T16:26:33.167 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-12T16:26:33.168 INFO:teuthology.orchestra.run.smithi149.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-12T16:26:33.168 INFO:teuthology.orchestra.run.smithi149.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-12T16:26:33.168 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:33.168 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:26:33.168 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:33.168 INFO:teuthology.orchestra.run.smithi149.stdout:Install 6 Packages 2024-02-12T16:26:33.168 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:33.169 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 3.0 M 2024-02-12T16:26:33.169 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 15 M 2024-02-12T16:26:33.169 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:26:33.398 INFO:teuthology.orchestra.run.smithi149.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 381 kB/s | 24 kB 00:00 2024-02-12T16:26:33.420 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:26:33.421 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:33.421 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:26:33.422 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:33.423 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-02-12T16:26:33.423 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:33.423 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 3.0 M 2024-02-12T16:26:33.423 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 15 M 2024-02-12T16:26:33.423 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:26:33.424 INFO:teuthology.orchestra.run.smithi149.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 101 kB/s | 8.9 kB 00:00 2024-02-12T16:26:33.490 INFO:teuthology.orchestra.run.smithi149.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 17 MB/s | 2.7 MB 00:00 2024-02-12T16:26:33.565 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 346 kB/s | 24 kB 00:00 2024-02-12T16:26:33.590 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 94 kB/s | 8.9 kB 00:00 2024-02-12T16:26:33.657 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 16 MB/s | 2.7 MB 00:00 2024-02-12T16:26:33.699 INFO:teuthology.orchestra.run.smithi149.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 142 kB/s | 43 kB 00:00 2024-02-12T16:26:33.816 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 149 kB/s | 34 kB 00:00 2024-02-12T16:26:33.899 INFO:teuthology.orchestra.run.smithi149.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 71 kB/s | 34 kB 00:00 2024-02-12T16:26:33.907 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 124 kB/s | 43 kB 00:00 2024-02-12T16:26:33.966 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 637 kB/s | 196 kB 00:00 2024-02-12T16:26:33.968 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:33.968 INFO:teuthology.orchestra.run.smithi104.stdout:Total 5.4 MB/s | 3.0 MB 00:00 2024-02-12T16:26:33.978 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:26:33.987 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:26:33.987 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:26:34.091 INFO:teuthology.orchestra.run.smithi149.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 327 kB/s | 196 kB 00:00 2024-02-12T16:26:34.094 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:34.094 INFO:teuthology.orchestra.run.smithi149.stdout:Total 3.2 MB/s | 3.0 MB 00:00 2024-02-12T16:26:34.103 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:26:34.113 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:26:34.113 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:26:34.167 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:26:34.169 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:26:34.283 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:26:34.286 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:26:34.475 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:26:34.575 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:26:34.698 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-12T16:26:34.778 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-12T16:26:35.001 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-12T16:26:35.023 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-12T16:26:35.202 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-12T16:26:35.224 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-12T16:26:35.348 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-12T16:26:35.378 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-12T16:26:36.354 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-12T16:26:36.379 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-12T16:26:36.629 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-12T16:26:36.671 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-12T16:26:36.922 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-12T16:26:36.922 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-12T16:26:36.922 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-12T16:26:36.922 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-12T16:26:36.922 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-12T16:26:36.922 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-12T16:26:36.969 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-12T16:26:36.969 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-12T16:26:36.969 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-12T16:26:36.969 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-12T16:26:36.969 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-12T16:26:36.969 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-12T16:26:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-12T16:26:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:37.301 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:26:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-12T16:26:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-12T16:26:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-12T16:26:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-12T16:26:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-12T16:26:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-12T16:26:37.301 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:37.301 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:26:37.371 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-12T16:26:37.371 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:37.371 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:26:37.371 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-12T16:26:37.371 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-12T16:26:37.371 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-12T16:26:37.372 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-12T16:26:37.372 INFO:teuthology.orchestra.run.smithi149.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-12T16:26:37.372 INFO:teuthology.orchestra.run.smithi149.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-12T16:26:37.372 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:37.372 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:26:37.468 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-12T16:26:37.493 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-12T16:26:38.220 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:31 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:26:38.265 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:30 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:26:39.765 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:26:39.766 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================ 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================ 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:39.767 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:26:39.768 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================ 2024-02-12T16:26:39.768 INFO:teuthology.orchestra.run.smithi149.stdout:Install 8 Packages 2024-02-12T16:26:39.768 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:39.768 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 37 M 2024-02-12T16:26:39.768 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 210 M 2024-02-12T16:26:39.768 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:26:39.784 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================ 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================ 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-12T16:26:39.786 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-12T16:26:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-12T16:26:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:39.787 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:26:39.787 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================ 2024-02-12T16:26:39.787 INFO:teuthology.orchestra.run.smithi104.stdout:Install 8 Packages 2024-02-12T16:26:39.787 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:39.787 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 37 M 2024-02-12T16:26:39.787 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 210 M 2024-02-12T16:26:39.787 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:26:40.022 INFO:teuthology.orchestra.run.smithi149.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 50 MB/s | 8.6 MB 00:00 2024-02-12T16:26:40.195 INFO:teuthology.orchestra.run.smithi104.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 49 MB/s | 8.6 MB 00:00 2024-02-12T16:26:40.205 INFO:teuthology.orchestra.run.smithi149.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-02-12T16:26:40.272 INFO:teuthology.orchestra.run.smithi149.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2024-02-12T16:26:40.297 INFO:teuthology.orchestra.run.smithi149.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-02-12T16:26:40.378 INFO:teuthology.orchestra.run.smithi104.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-02-12T16:26:40.429 INFO:teuthology.orchestra.run.smithi104.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2024-02-12T16:26:40.823 INFO:teuthology.orchestra.run.smithi149.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.7 MB/s | 3.7 MB 00:00 2024-02-12T16:26:40.856 INFO:teuthology.orchestra.run.smithi149.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2024-02-12T16:26:40.887 INFO:teuthology.orchestra.run.smithi104.stdout:(4/8): python3-scipy-1.0.0-21.module_el8+762+77 31 MB/s | 14 MB 00:00 2024-02-12T16:26:40.996 INFO:teuthology.orchestra.run.smithi104.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.7 MB/s | 3.7 MB 00:00 2024-02-12T16:26:41.032 INFO:teuthology.orchestra.run.smithi149.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 19 MB/s | 14 MB 00:00 2024-02-12T16:26:41.055 INFO:teuthology.orchestra.run.smithi104.stdout:(6/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.7 MB/s | 4.9 MB 00:01 2024-02-12T16:26:41.165 INFO:teuthology.orchestra.run.smithi149.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 504 kB/s | 172 kB 00:00 2024-02-12T16:26:41.165 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:41.166 INFO:teuthology.orchestra.run.smithi149.stdout:Total 27 MB/s | 37 MB 00:01 2024-02-12T16:26:41.347 INFO:teuthology.orchestra.run.smithi104.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 493 kB/s | 172 kB 00:00 2024-02-12T16:26:41.395 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:26:41.397 INFO:teuthology.orchestra.run.smithi104.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2024-02-12T16:26:41.397 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:41.397 INFO:teuthology.orchestra.run.smithi104.stdout:Total 23 MB/s | 37 MB 00:01 2024-02-12T16:26:41.414 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:26:41.414 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:26:41.630 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:26:41.648 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:26:41.648 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:26:41.754 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:26:41.758 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:26:41.969 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:26:41.972 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:26:42.355 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:26:42.513 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-12T16:26:42.634 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:26:42.635 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-12T16:26:42.748 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-12T16:26:42.779 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-12T16:26:42.886 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-12T16:26:43.048 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-12T16:26:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-12T16:26:43.537 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:26:43.648 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-12T16:26:44.300 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:26:44.479 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-12T16:26:44.681 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:26:45.195 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-12T16:26:45.310 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:26:45.514 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-12T16:26:45.523 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-12T16:26:45.796 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-12T16:26:46.192 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-12T16:26:46.517 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-12T16:26:46.756 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-12T16:26:47.562 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-12T16:26:48.504 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-12T16:26:48.834 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-12T16:26:49.019 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-12T16:26:49.358 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-12T16:26:49.358 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-12T16:26:49.358 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-12T16:26:49.358 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:26:49.358 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-12T16:26:49.358 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-12T16:26:49.358 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-12T16:26:49.358 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-12T16:26:49.815 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-12T16:26:50.181 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:50.182 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:26:50.238 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-12T16:26:50.298 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-rook 2024-02-12T16:26:50.573 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-12T16:26:50.573 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-12T16:26:50.573 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-12T16:26:50.573 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-12T16:26:50.574 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-12T16:26:50.574 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-12T16:26:50.574 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-12T16:26:50.574 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-12T16:26:50.983 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-12T16:26:50.983 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:50.983 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:26:50.983 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-12T16:26:50.983 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-12T16:26:50.984 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-12T16:26:50.984 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-12T16:26:50.984 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-12T16:26:50.984 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-12T16:26:50.984 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-12T16:26:50.984 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-12T16:26:50.984 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:50.984 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:26:51.069 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:43 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:26:51.117 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-rook 2024-02-12T16:26:51.866 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:44 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:26:52.587 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:26:52.589 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:52.589 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-02-12T16:26:52.589 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:52.589 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:26:52.589 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-12T16:26:52.589 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-02-12T16:26:52.589 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-12T16:26:52.589 INFO:teuthology.orchestra.run.smithi149.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-12T16:26:52.589 INFO:teuthology.orchestra.run.smithi149.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout:Install 13 Packages 2024-02-12T16:26:52.590 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:52.591 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 1.9 M 2024-02-12T16:26:52.591 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 24 M 2024-02-12T16:26:52.591 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:26:53.358 INFO:teuthology.orchestra.run.smithi149.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 581 kB/s | 27 kB 00:00 2024-02-12T16:26:53.387 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:26:53.392 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:53.392 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-12T16:26:53.392 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:53.392 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:26:53.392 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-12T16:26:53.392 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-12T16:26:53.392 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-12T16:26:53.392 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:26:53.393 INFO:teuthology.orchestra.run.smithi104.stdout:Install 13 Packages 2024-02-12T16:26:53.394 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:26:53.394 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 1.9 M 2024-02-12T16:26:53.394 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 24 M 2024-02-12T16:26:53.394 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:26:53.408 INFO:teuthology.orchestra.run.smithi149.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 207 kB/s | 20 kB 00:00 2024-02-12T16:26:53.433 INFO:teuthology.orchestra.run.smithi149.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 218 kB/s | 27 kB 00:00 2024-02-12T16:26:53.459 INFO:teuthology.orchestra.run.smithi149.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2024-02-12T16:26:53.484 INFO:teuthology.orchestra.run.smithi149.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.4 MB/s | 110 kB 00:00 2024-02-12T16:26:53.593 INFO:teuthology.orchestra.run.smithi149.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 304 kB/s | 33 kB 00:00 2024-02-12T16:26:53.643 INFO:teuthology.orchestra.run.smithi149.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 326 kB/s | 16 kB 00:00 2024-02-12T16:26:53.676 INFO:teuthology.orchestra.run.smithi149.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 196 kB/s | 43 kB 00:00 2024-02-12T16:26:53.718 INFO:teuthology.orchestra.run.smithi149.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 1.1 MB/s | 81 kB 00:00 2024-02-12T16:26:53.769 INFO:teuthology.orchestra.run.smithi149.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 462 kB/s | 155 kB 00:00 2024-02-12T16:26:53.794 INFO:teuthology.orchestra.run.smithi149.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 909 kB/s | 64 kB 00:00 2024-02-12T16:26:53.903 INFO:teuthology.orchestra.run.smithi149.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 460 kB/s | 61 kB 00:00 2024-02-12T16:26:53.986 INFO:teuthology.orchestra.run.smithi149.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.8 MB/s | 1.2 MB 00:00 2024-02-12T16:26:53.987 INFO:teuthology.orchestra.run.smithi104.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 662 kB/s | 27 kB 00:00 2024-02-12T16:26:53.989 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:53.989 INFO:teuthology.orchestra.run.smithi149.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-02-12T16:26:54.046 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:26:54.053 INFO:teuthology.orchestra.run.smithi104.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 247 kB/s | 27 kB 00:00 2024-02-12T16:26:54.061 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:26:54.061 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:26:54.078 INFO:teuthology.orchestra.run.smithi104.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 151 kB/s | 20 kB 00:00 2024-02-12T16:26:54.104 INFO:teuthology.orchestra.run.smithi104.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2024-02-12T16:26:54.154 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:26:54.157 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:26:54.171 INFO:teuthology.orchestra.run.smithi104.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.2 MB/s | 110 kB 00:00 2024-02-12T16:26:54.421 INFO:teuthology.orchestra.run.smithi104.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 132 kB/s | 33 kB 00:00 2024-02-12T16:26:54.446 INFO:teuthology.orchestra.run.smithi104.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 124 kB/s | 43 kB 00:00 2024-02-12T16:26:54.522 INFO:teuthology.orchestra.run.smithi104.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 350 kB/s | 155 kB 00:00 2024-02-12T16:26:54.555 INFO:teuthology.orchestra.run.smithi104.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2024-02-12T16:26:54.655 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:26:54.731 INFO:teuthology.orchestra.run.smithi104.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 367 kB/s | 64 kB 00:00 2024-02-12T16:26:54.764 INFO:teuthology.orchestra.run.smithi104.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 256 kB/s | 81 kB 00:00 2024-02-12T16:26:54.863 INFO:teuthology.orchestra.run.smithi104.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 463 kB/s | 61 kB 00:00 2024-02-12T16:26:54.972 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-12T16:26:55.148 INFO:teuthology.orchestra.run.smithi104.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.9 MB/s | 1.2 MB 00:00 2024-02-12T16:26:55.148 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:26:55.149 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2024-02-12T16:26:55.189 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-12T16:26:55.204 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:26:55.220 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:26:55.220 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:26:55.308 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:26:55.311 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:26:55.405 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-12T16:26:55.626 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-12T16:26:55.825 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-12T16:26:55.938 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:26:56.064 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-12T16:26:56.181 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-12T16:26:56.382 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-12T16:26:56.424 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-12T16:26:56.606 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-12T16:26:56.689 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-12T16:26:56.927 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-12T16:26:57.142 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-12T16:26:57.407 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-12T16:26:57.645 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-12T16:26:57.732 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-12T16:26:57.916 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-12T16:26:57.957 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-12T16:26:58.157 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-12T16:26:58.359 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-12T16:26:58.560 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-12T16:26:58.877 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-12T16:26:58.877 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-12T16:26:58.877 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-12T16:26:58.877 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-12T16:26:58.877 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-12T16:26:58.877 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-12T16:26:58.877 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-12T16:26:58.877 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-12T16:26:58.878 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-12T16:26:58.878 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-12T16:26:58.878 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-12T16:26:58.878 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-12T16:26:58.878 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-12T16:26:58.999 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-12T16:26:59.276 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-12T16:26:59.339 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-12T16:26:59.339 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:59.339 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:26:59.339 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-12T16:26:59.339 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:26:59.340 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:26:59.454 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-cephadm 2024-02-12T16:26:59.525 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-12T16:26:59.752 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-12T16:26:59.911 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-12T16:27:00.208 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-12T16:27:00.208 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-12T16:27:00.208 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-12T16:27:00.208 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-12T16:27:00.209 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-12T16:27:00.209 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-12T16:27:00.209 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-12T16:27:00.209 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-12T16:27:00.209 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-12T16:27:00.209 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-12T16:27:00.209 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-12T16:27:00.209 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-12T16:27:00.209 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-12T16:27:00.230 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:52 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:00.658 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-12T16:27:00.658 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-12T16:27:00.659 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-12T16:27:00.660 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-12T16:27:00.660 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-12T16:27:00.660 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:00.660 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:00.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-cephadm 2024-02-12T16:27:01.539 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:54 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:01.747 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:01.748 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:01.748 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-02-12T16:27:01.748 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:01.748 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:27:01.748 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-12T16:27:01.749 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-02-12T16:27:01.749 INFO:teuthology.orchestra.run.smithi149.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-12T16:27:01.749 INFO:teuthology.orchestra.run.smithi149.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-12T16:27:01.749 INFO:teuthology.orchestra.run.smithi149.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-12T16:27:01.749 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-12T16:27:01.749 INFO:teuthology.orchestra.run.smithi149.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-12T16:27:01.749 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:01.749 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:27:01.749 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:01.749 INFO:teuthology.orchestra.run.smithi149.stdout:Install 6 Packages 2024-02-12T16:27:01.749 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:01.750 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 5.8 M 2024-02-12T16:27:01.750 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 24 M 2024-02-12T16:27:01.750 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:27:02.178 INFO:teuthology.orchestra.run.smithi149.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.1 MB/s | 89 kB 00:00 2024-02-12T16:27:02.303 INFO:teuthology.orchestra.run.smithi149.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 170 kB/s | 21 kB 00:00 2024-02-12T16:27:02.587 INFO:teuthology.orchestra.run.smithi149.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.2 MB/s | 538 kB 00:00 2024-02-12T16:27:02.612 INFO:teuthology.orchestra.run.smithi149.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.1 MB/s | 355 kB 00:00 2024-02-12T16:27:02.646 INFO:teuthology.orchestra.run.smithi149.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 834 kB/s | 49 kB 00:00 2024-02-12T16:27:03.047 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:27:03.049 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:03.050 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-02-12T16:27:03.050 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:03.050 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 5.8 M 2024-02-12T16:27:03.050 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 24 M 2024-02-12T16:27:03.050 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:27:03.196 INFO:teuthology.orchestra.run.smithi149.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 4.5 MB/s | 4.8 MB 00:01 2024-02-12T16:27:03.197 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:03.197 INFO:teuthology.orchestra.run.smithi149.stdout:Total 4.0 MB/s | 5.8 MB 00:01 2024-02-12T16:27:03.238 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 89 kB 00:00 2024-02-12T16:27:03.249 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:27:03.259 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:27:03.259 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:27:03.360 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:27:03.364 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:27:03.554 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.5 MB/s | 538 kB 00:00 2024-02-12T16:27:03.580 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 62 kB/s | 21 kB 00:00 2024-02-12T16:27:03.639 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:27:03.730 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 326 kB/s | 49 kB 00:00 2024-02-12T16:27:03.892 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-12T16:27:03.972 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 852 kB/s | 355 kB 00:00 2024-02-12T16:27:04.072 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.4 MB/s | 4.8 MB 00:00 2024-02-12T16:27:04.074 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:04.074 INFO:teuthology.orchestra.run.smithi104.stdout:Total 5.7 MB/s | 5.8 MB 00:01 2024-02-12T16:27:04.126 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:27:04.135 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:27:04.135 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:27:04.235 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:27:04.238 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:27:04.270 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-12T16:27:04.511 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:27:04.818 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-12T16:27:05.146 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-12T16:27:05.190 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-12T16:27:05.572 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-12T16:27:05.797 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-12T16:27:05.967 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-12T16:27:06.106 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-12T16:27:06.269 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-12T16:27:06.269 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-12T16:27:06.269 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-12T16:27:06.269 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-12T16:27:06.269 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-12T16:27:06.269 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-12T16:27:06.488 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-12T16:27:06.678 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-12T16:27:06.678 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:06.678 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:27:06.678 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-12T16:27:06.678 INFO:teuthology.orchestra.run.smithi149.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-12T16:27:06.678 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-12T16:27:06.678 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:06.678 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:06.748 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-12T16:27:06.819 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-fuse 2024-02-12T16:27:06.959 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-12T16:27:07.273 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-12T16:27:07.274 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-12T16:27:07.274 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-12T16:27:07.274 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-12T16:27:07.274 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-12T16:27:07.274 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-12T16:27:07.588 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:59 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-12T16:27:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:07.696 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:27:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-12T16:27:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-12T16:27:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-12T16:27:07.697 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:07.697 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:07.852 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-fuse 2024-02-12T16:27:08.603 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:01 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:09.112 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:09.112 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:09.112 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:09.112 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:09.112 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:27:09.112 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-12T16:27:09.112 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-02-12T16:27:09.112 INFO:teuthology.orchestra.run.smithi149.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-12T16:27:09.113 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:09.113 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:27:09.113 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:09.113 INFO:teuthology.orchestra.run.smithi149.stdout:Install 2 Packages 2024-02-12T16:27:09.113 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:09.113 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 968 k 2024-02-12T16:27:09.113 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 3.2 M 2024-02-12T16:27:09.113 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:27:09.839 INFO:teuthology.orchestra.run.smithi149.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2024-02-12T16:27:10.106 INFO:teuthology.orchestra.run.smithi149.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 252 kB/s | 83 kB 00:00 2024-02-12T16:27:10.107 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:10.107 INFO:teuthology.orchestra.run.smithi149.stdout:Total 973 kB/s | 968 kB 00:00 2024-02-12T16:27:10.116 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:10.117 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:10.117 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:10.117 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:10.117 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:27:10.117 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-12T16:27:10.117 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-12T16:27:10.117 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-12T16:27:10.117 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:10.117 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:27:10.117 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:10.118 INFO:teuthology.orchestra.run.smithi104.stdout:Install 2 Packages 2024-02-12T16:27:10.118 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:10.118 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 968 k 2024-02-12T16:27:10.118 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 3.2 M 2024-02-12T16:27:10.118 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:27:10.119 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:27:10.141 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:27:10.141 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:27:10.236 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:27:10.239 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:27:10.381 INFO:teuthology.orchestra.run.smithi104.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-02-12T16:27:10.631 INFO:teuthology.orchestra.run.smithi104.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 270 kB/s | 83 kB 00:00 2024-02-12T16:27:10.631 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:10.632 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.8 MB/s | 968 kB 00:00 2024-02-12T16:27:10.641 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:27:10.662 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:27:10.662 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:27:10.749 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:27:10.753 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:27:10.848 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:27:11.072 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:27:11.091 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-12T16:27:11.300 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-12T16:27:11.342 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-12T16:27:11.500 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-12T16:27:11.690 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-12T16:27:11.690 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-12T16:27:11.901 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-12T16:27:11.901 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-12T16:27:12.094 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-12T16:27:12.095 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:12.095 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:27:12.095 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-12T16:27:12.095 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:12.095 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:12.270 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-12T16:27:12.270 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:12.270 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:27:12.270 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-12T16:27:12.270 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:12.270 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:12.272 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install librados-devel 2024-02-12T16:27:12.497 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados-devel 2024-02-12T16:27:13.055 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:05 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:13.255 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:06 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:14.583 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:14.584 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:14.584 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:14.584 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:14.584 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:27:14.584 INFO:teuthology.orchestra.run.smithi149.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-12T16:27:14.584 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:14.584 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:27:14.585 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:14.585 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-02-12T16:27:14.585 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:14.585 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 131 k 2024-02-12T16:27:14.585 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 439 k 2024-02-12T16:27:14.585 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:27:14.637 INFO:teuthology.orchestra.run.smithi149.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.5 MB/s | 131 kB 00:00 2024-02-12T16:27:14.637 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:14.638 INFO:teuthology.orchestra.run.smithi149.stdout:Total 2.4 MB/s | 131 kB 00:00 2024-02-12T16:27:14.638 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:27:14.643 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:27:14.643 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:27:14.692 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:27:14.695 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:27:14.755 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:14.756 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:14.756 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:14.756 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:14.756 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:27:14.756 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-12T16:27:14.757 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:14.757 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:27:14.757 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:14.757 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-12T16:27:14.757 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:14.757 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 131 k 2024-02-12T16:27:14.757 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 439 k 2024-02-12T16:27:14.757 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:27:14.801 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:27:14.802 INFO:teuthology.orchestra.run.smithi104.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 131 kB 00:00 2024-02-12T16:27:14.803 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:14.803 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.8 MB/s | 131 kB 00:00 2024-02-12T16:27:14.803 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:27:14.808 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:27:14.808 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:27:14.857 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:27:14.860 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:27:14.967 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:27:15.048 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:15.223 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:15.349 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:15.522 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:15.753 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:15.753 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:15.753 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:27:15.753 INFO:teuthology.orchestra.run.smithi149.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-12T16:27:15.753 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:15.753 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:15.902 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:15.903 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:15.903 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:27:15.903 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-12T16:27:15.903 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:15.903 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:16.109 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs2 2024-02-12T16:27:16.868 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:09 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:17.095 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install libcephfs2 2024-02-12T16:27:17.876 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:09 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:18.324 INFO:teuthology.orchestra.run.smithi104.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:18.369 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:18.371 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-12T16:27:18.371 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:18.518 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs-devel 2024-02-12T16:27:19.266 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:12 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:19.357 INFO:teuthology.orchestra.run.smithi149.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:19.402 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:19.403 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-02-12T16:27:19.404 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:19.495 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install libcephfs-devel 2024-02-12T16:27:20.261 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:12 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:20.772 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:20.776 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:20.776 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:20.776 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:20.777 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:27:20.777 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-12T16:27:20.777 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:20.777 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:27:20.777 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:20.777 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-12T16:27:20.777 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:20.777 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 24 k 2024-02-12T16:27:20.777 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 94 k 2024-02-12T16:27:20.777 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:27:20.842 INFO:teuthology.orchestra.run.smithi104.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 358 kB/s | 24 kB 00:00 2024-02-12T16:27:20.842 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:20.842 INFO:teuthology.orchestra.run.smithi104.stdout:Total 352 kB/s | 24 kB 00:00 2024-02-12T16:27:20.842 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:27:20.845 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:27:20.846 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:27:20.848 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:27:20.851 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:27:20.907 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:27:21.113 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:21.418 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:21.767 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 24 k 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 94 k 2024-02-12T16:27:21.768 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:27:21.774 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:21.774 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:21.774 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:27:21.774 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-12T16:27:21.774 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:21.774 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:21.813 INFO:teuthology.orchestra.run.smithi149.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 546 kB/s | 24 kB 00:00 2024-02-12T16:27:21.814 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:21.814 INFO:teuthology.orchestra.run.smithi149.stdout:Total 529 kB/s | 24 kB 00:00 2024-02-12T16:27:21.814 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:27:21.818 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:27:21.818 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:27:21.820 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:27:21.823 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:27:21.875 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:27:21.896 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados2 2024-02-12T16:27:22.080 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:22.365 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:22.646 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:15 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:22.723 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:22.724 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:22.724 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:27:22.724 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-12T16:27:22.724 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:22.724 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:22.858 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install librados2 2024-02-12T16:27:23.637 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:15 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:24.104 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:24.150 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:24.151 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-12T16:27:24.151 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:24.262 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librbd1 2024-02-12T16:27:25.020 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:18 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:25.106 INFO:teuthology.orchestra.run.smithi149.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:25.152 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:25.153 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-02-12T16:27:25.153 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:25.249 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install librbd1 2024-02-12T16:27:26.020 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:18 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:26.480 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:26.527 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:26.528 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-12T16:27:26.528 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:26.641 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rados 2024-02-12T16:27:27.391 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:20 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:27.492 INFO:teuthology.orchestra.run.smithi149.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:27.538 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:27.540 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-02-12T16:27:27.540 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:27.640 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-rados 2024-02-12T16:27:28.411 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:20 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:28.854 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:28.899 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:28.901 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-12T16:27:28.901 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:29.135 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rgw 2024-02-12T16:27:29.890 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:22 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:29.891 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:29.937 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:29.938 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-02-12T16:27:29.938 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:30.039 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-rgw 2024-02-12T16:27:30.806 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:22 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:31.351 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:31.397 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:31.398 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-12T16:27:31.398 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:31.510 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-cephfs 2024-02-12T16:27:32.244 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:32.266 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:25 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:32.290 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:32.291 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-02-12T16:27:32.292 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:32.382 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-cephfs 2024-02-12T16:27:33.156 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:25 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:33.709 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:33.754 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:33.755 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-12T16:27:33.755 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:33.874 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rbd 2024-02-12T16:27:34.598 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:34.623 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:27 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:34.645 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:34.646 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-02-12T16:27:34.646 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:34.742 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-rbd 2024-02-12T16:27:35.512 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:27 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:36.083 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:36.128 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:36.129 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-12T16:27:36.129 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:36.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-fuse 2024-02-12T16:27:36.973 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-12T16:27:37.005 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:30 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:37.019 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:37.020 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-02-12T16:27:37.020 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:37.119 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install rbd-fuse 2024-02-12T16:27:37.896 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:29 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:38.517 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 89 k 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 298 k 2024-02-12T16:27:38.518 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:27:38.562 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2024-02-12T16:27:38.562 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:38.563 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.0 MB/s | 89 kB 00:00 2024-02-12T16:27:38.563 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:27:38.572 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:27:38.572 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:27:38.623 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:27:38.626 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:27:38.732 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:27:38.951 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:39.243 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:39.419 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:39.419 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:39.420 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:39.420 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:39.420 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:27:39.420 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-12T16:27:39.420 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:39.420 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:27:39.420 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:39.420 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-02-12T16:27:39.420 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:39.421 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 89 k 2024-02-12T16:27:39.421 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 298 k 2024-02-12T16:27:39.421 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:27:39.678 INFO:teuthology.orchestra.run.smithi149.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 346 kB/s | 89 kB 00:00 2024-02-12T16:27:39.678 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:39.679 INFO:teuthology.orchestra.run.smithi149.stdout:Total 344 kB/s | 89 kB 00:00 2024-02-12T16:27:39.679 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:27:39.689 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:27:39.689 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:27:39.739 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:27:39.742 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:27:39.850 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:27:39.938 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:39.938 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:39.938 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:27:39.938 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-12T16:27:39.938 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:39.938 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:40.078 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:40.290 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-mirror 2024-02-12T16:27:40.386 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:40.764 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:40.764 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:40.764 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:27:40.765 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-12T16:27:40.765 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:40.765 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:40.984 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install rbd-mirror 2024-02-12T16:27:41.055 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:34 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:41.761 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:33 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:42.562 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:42.562 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:42.562 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:42.563 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:42.563 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:27:42.563 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-12T16:27:42.563 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:42.563 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:27:42.563 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:42.563 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-12T16:27:42.563 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:42.564 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 3.7 M 2024-02-12T16:27:42.564 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 18 M 2024-02-12T16:27:42.564 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:27:42.666 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 36 MB/s | 3.7 MB 00:00 2024-02-12T16:27:42.667 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:42.667 INFO:teuthology.orchestra.run.smithi104.stdout:Total 36 MB/s | 3.7 MB 00:00 2024-02-12T16:27:42.668 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:27:42.679 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:27:42.679 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:27:42.741 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:27:42.744 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:27:43.239 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:27:43.283 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:43.284 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:43.284 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:43.284 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:43.284 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:27:43.284 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-12T16:27:43.284 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:43.284 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:27:43.284 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:43.284 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-02-12T16:27:43.284 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:43.285 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 3.7 M 2024-02-12T16:27:43.285 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 18 M 2024-02-12T16:27:43.285 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:27:43.374 INFO:teuthology.orchestra.run.smithi149.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2024-02-12T16:27:43.375 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:43.375 INFO:teuthology.orchestra.run.smithi149.stdout:Total 41 MB/s | 3.7 MB 00:00 2024-02-12T16:27:43.375 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:27:43.387 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:27:43.387 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:27:43.435 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:43.450 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:27:43.454 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:27:43.862 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:43.942 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:27:44.153 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:44.264 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:44.264 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:44.264 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:27:44.265 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-12T16:27:44.265 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:44.265 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:44.486 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-nbd 2024-02-12T16:27:44.577 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:45.239 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:45.239 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:45.239 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:27:45.239 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-12T16:27:45.239 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:45.239 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:45.245 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:38 ago on Mon 12 Feb 2024 04:25:07 PM UTC. 2024-02-12T16:27:45.456 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install rbd-nbd 2024-02-12T16:27:46.230 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:02:38 ago on Mon 12 Feb 2024 04:25:08 PM UTC. 2024-02-12T16:27:46.747 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-12T16:27:46.748 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:46.748 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:46.748 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:46.748 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-12T16:27:46.748 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-12T16:27:46.748 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:46.748 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-12T16:27:46.748 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-12T16:27:46.748 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-12T16:27:46.748 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:46.749 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 173 k 2024-02-12T16:27:46.749 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 649 k 2024-02-12T16:27:46.749 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-12T16:27:46.785 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.7 MB/s | 173 kB 00:00 2024-02-12T16:27:46.786 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:46.786 INFO:teuthology.orchestra.run.smithi104.stdout:Total 4.5 MB/s | 173 kB 00:00 2024-02-12T16:27:46.786 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-12T16:27:46.797 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-12T16:27:46.797 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-12T16:27:46.847 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-12T16:27:46.850 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-12T16:27:46.972 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-12T16:27:47.158 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:47.430 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:47.761 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-02-12T16:27:47.761 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:47.761 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-02-12T16:27:47.761 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:47.762 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-02-12T16:27:47.762 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-12T16:27:47.762 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:47.762 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-02-12T16:27:47.762 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-02-12T16:27:47.762 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-02-12T16:27:47.762 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:47.762 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 173 k 2024-02-12T16:27:47.763 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 649 k 2024-02-12T16:27:47.763 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-02-12T16:27:47.792 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:47.792 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:47.792 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-12T16:27:47.792 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-12T16:27:47.792 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-12T16:27:47.792 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-12T16:27:47.810 INFO:teuthology.orchestra.run.smithi149.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 173 kB 00:00 2024-02-12T16:27:47.811 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-02-12T16:27:47.811 INFO:teuthology.orchestra.run.smithi149.stdout:Total 3.5 MB/s | 173 kB 00:00 2024-02-12T16:27:47.811 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-02-12T16:27:47.822 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-02-12T16:27:47.822 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-02-12T16:27:47.872 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-02-12T16:27:47.875 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-02-12T16:27:47.984 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-02-12T16:27:48.065 DEBUG:teuthology.parallel:result is None 2024-02-12T16:27:48.201 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:48.539 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:48.944 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-12T16:27:48.944 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:48.945 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-02-12T16:27:48.945 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-12T16:27:48.945 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:27:48.945 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-02-12T16:27:49.122 DEBUG:teuthology.parallel:result is None 2024-02-12T16:27:49.122 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-12T16:27:49.122 INFO:teuthology.packaging:ref: None 2024-02-12T16:27:49.122 INFO:teuthology.packaging:tag: v16.2.4 2024-02-12T16:27:49.122 INFO:teuthology.packaging:branch: None 2024-02-12T16:27:49.122 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:27:49.123 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-12T16:27:49.259 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-12T16:27:49.283 INFO:teuthology.orchestra.run.smithi104.stdout:16.2.4-0.el8 2024-02-12T16:27:49.283 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-12T16:27:49.283 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-12T16:27:49.285 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-12T16:27:49.285 INFO:teuthology.packaging:ref: None 2024-02-12T16:27:49.285 INFO:teuthology.packaging:tag: v16.2.4 2024-02-12T16:27:49.285 INFO:teuthology.packaging:branch: None 2024-02-12T16:27:49.285 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T16:27:49.285 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-12T16:27:49.653 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-12T16:27:49.676 INFO:teuthology.orchestra.run.smithi149.stdout:16.2.4-0.el8 2024-02-12T16:27:49.676 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-12T16:27:49.676 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-12T16:27:49.678 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-12T16:27:49.678 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-12T16:27:49.678 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-12T16:27:49.712 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-12T16:27:49.712 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-12T16:27:49.745 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-12T16:27:49.745 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-12T16:27:49.745 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-02-12T16:27:49.782 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-12T16:27:49.851 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-12T16:27:49.852 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/daemon-helper 2024-02-12T16:27:49.878 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-12T16:27:49.944 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-12T16:27:49.944 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-12T16:27:49.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-12T16:27:49.973 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-12T16:27:50.042 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-12T16:27:50.042 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-12T16:27:50.068 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-12T16:27:50.135 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-12T16:27:50.135 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-12T16:27:50.135 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-02-12T16:27:50.163 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-12T16:27:50.230 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-12T16:27:50.230 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/stdin-killer 2024-02-12T16:27:50.256 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-12T16:27:50.323 INFO:teuthology.run_tasks:Running task print... 2024-02-12T16:27:50.333 INFO:teuthology.task.print:**** done install task... 2024-02-12T16:27:50.333 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-12T16:27:50.402 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-12T16:27:50.403 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-12T16:27:50.403 INFO:tasks.cephadm:Cluster fsid is aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:27:50.403 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-12T16:27:50.403 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-12T16:27:50.403 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi104': '172.21.15.104', 'mon.smithi149': '172.21.15.149'} 2024-02-12T16:27:50.404 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-12T16:27:50.404 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2024-02-12T16:27:50.431 DEBUG:teuthology.orchestra.run.smithi149:> sudo hostname $(hostname -s) 2024-02-12T16:27:50.459 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-12T16:27:50.460 DEBUG:teuthology.orchestra.run.smithi104:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-12T16:27:50.798 INFO:teuthology.orchestra.run.smithi104.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 12 16:27 /home/ubuntu/cephtest/cephadm 2024-02-12T16:27:50.799 DEBUG:teuthology.orchestra.run.smithi149:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-12T16:27:51.019 INFO:teuthology.orchestra.run.smithi149.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 12 16:27 /home/ubuntu/cephtest/cephadm 2024-02-12T16:27:51.020 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-12T16:27:51.040 DEBUG:teuthology.orchestra.run.smithi149:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-12T16:27:51.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2024-02-12T16:27:51.109 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /etc/ceph 2024-02-12T16:27:51.135 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2024-02-12T16:27:51.176 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 777 /etc/ceph 2024-02-12T16:27:51.202 INFO:tasks.cephadm:Writing seed config... 2024-02-12T16:27:51.203 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-12T16:27:51.203 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-12T16:27:51.204 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-12T16:27:51.205 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-12T16:27:51.206 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-12T16:27:51.206 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-12T16:27:51.206 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-12T16:27:51.206 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-12T16:27:51.206 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-12T16:27:51.206 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-12T16:27:51.206 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-12T16:27:51.206 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-12T16:27:51.206 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-12T16:27:51.207 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-12T16:27:51.235 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = aa0fd796-c9c3-11ee-95b9-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-12T16:27:51.236 DEBUG:teuthology.orchestra.run.smithi104:mon.smithi104> sudo journalctl -f -n 0 -u ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104.service 2024-02-12T16:27:51.277 INFO:tasks.cephadm:Bootstrapping... 2024-02-12T16:27:51.277 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.104 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:27:51.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:-- Logs begin at Mon 2024-02-12 16:18:13 UTC. -- 2024-02-12T16:27:51.552 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-12T16:27:52.553 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: 4.8.3 2024-02-12T16:27:52.563 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying podman|docker is present... 2024-02-12T16:27:52.563 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-12T16:27:52.586 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: 4.8.3 2024-02-12T16:27:52.599 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying lvm2 is present... 2024-02-12T16:27:52.599 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying time synchronization is in place... 2024-02-12T16:27:52.599 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-enabled chrony.service 2024-02-12T16:27:52.606 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-12T16:27:52.607 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-active chrony.service 2024-02-12T16:27:52.613 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-02-12T16:27:52.613 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-12T16:27:52.619 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: enabled 2024-02-12T16:27:52.620 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-active chronyd.service 2024-02-12T16:27:52.625 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: active 2024-02-12T16:27:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:Unit chronyd.service is enabled and running 2024-02-12T16:27:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:Repeating the final host check... 2024-02-12T16:27:52.626 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-12T16:27:52.646 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: 4.8.3 2024-02-12T16:27:52.659 INFO:teuthology.orchestra.run.smithi104.stderr:podman|docker (/bin/podman) is present 2024-02-12T16:27:52.659 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl is present 2024-02-12T16:27:52.659 INFO:teuthology.orchestra.run.smithi104.stderr:lvcreate is present 2024-02-12T16:27:52.659 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-enabled chrony.service 2024-02-12T16:27:52.664 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-12T16:27:52.665 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-active chrony.service 2024-02-12T16:27:52.670 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-02-12T16:27:52.671 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-12T16:27:52.677 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: enabled 2024-02-12T16:27:52.677 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-active chronyd.service 2024-02-12T16:27:52.682 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: active 2024-02-12T16:27:52.683 INFO:teuthology.orchestra.run.smithi104.stderr:Unit chronyd.service is enabled and running 2024-02-12T16:27:52.683 INFO:teuthology.orchestra.run.smithi104.stderr:Host looks OK 2024-02-12T16:27:52.683 INFO:teuthology.orchestra.run.smithi104.stderr:Cluster fsid: aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:27:52.683 INFO:teuthology.orchestra.run.smithi104.stderr:Acquiring lock 139836925118952 on /run/cephadm/aa0fd796-c9c3-11ee-95b9-87774f69a715.lock 2024-02-12T16:27:52.684 INFO:teuthology.orchestra.run.smithi104.stderr:Lock 139836925118952 acquired on /run/cephadm/aa0fd796-c9c3-11ee-95b9-87774f69a715.lock 2024-02-12T16:27:52.684 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying IP 172.21.15.104 port 3300 ... 2024-02-12T16:27:52.685 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying IP 172.21.15.104 port 6789 ... 2024-02-12T16:27:52.685 INFO:teuthology.orchestra.run.smithi104.stderr:Base mon IP is 172.21.15.104, final addrv is [v2:172.21.15.104:3300,v1:172.21.15.104:6789] 2024-02-12T16:27:52.685 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /sbin/ip route ls 2024-02-12T16:27:52.687 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.104 metric 100 2024-02-12T16:27:52.687 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.104 metric 100 2024-02-12T16:27:52.687 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /sbin/ip -6 route ls 2024-02-12T16:27:52.690 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-12T16:27:52.690 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-12T16:27:52.690 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1731sec hoplimit 64 pref medium 2024-02-12T16:27:52.690 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /sbin/ip -6 addr ls 2024-02-12T16:27:52.693 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-12T16:27:52.693 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-12T16:27:52.693 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-12T16:27:52.693 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-12T16:27:52.693 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:ce89/64 scope link 2024-02-12T16:27:52.693 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-12T16:27:52.694 INFO:teuthology.orchestra.run.smithi104.stderr:Mon IP 172.21.15.104 is in CIDR network 172.21.0.0/20 2024-02-12T16:27:52.694 INFO:teuthology.orchestra.run.smithi104.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-12T16:27:52.695 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-12T16:27:52.695 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-12T16:27:52.795 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-12T16:27:59.346 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Getting image source signatures 2024-02-12T16:27:59.356 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-12T16:27:59.356 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-12T16:28:11.616 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-12T16:28:11.758 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Writing manifest to image destination 2024-02-12T16:28:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-12T16:28:12.099 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-12T16:28:12.799 INFO:teuthology.orchestra.run.smithi104.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-12T16:28:13.898 INFO:teuthology.orchestra.run.smithi104.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-12T16:28:13.898 INFO:teuthology.orchestra.run.smithi104.stderr:Extracting ceph user uid/gid from container image... 2024-02-12T16:28:13.899 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-12T16:28:14.496 INFO:teuthology.orchestra.run.smithi104.stderr:stat: 167 167 2024-02-12T16:28:15.072 INFO:teuthology.orchestra.run.smithi104.stderr:Creating initial keys... 2024-02-12T16:28:15.072 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-12T16:28:15.651 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-authtool: AQAfR8plW8wMIhAA58nTgZAk5K+OBUBPZkV15g== 2024-02-12T16:28:16.220 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-12T16:28:16.766 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-authtool: AQAgR8plcf/QJhAALZg73OxHg8f8zYP3SsFdbQ== 2024-02-12T16:28:17.400 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-12T16:28:17.987 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-authtool: AQAhR8plCa3xMxAA4gQvvt4J6XJv/e3EUVg6ag== 2024-02-12T16:28:18.516 INFO:teuthology.orchestra.run.smithi104.stderr:Creating initial monmap... 2024-02-12T16:28:18.516 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpntmks2hq:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 --addv smithi104 [v2:172.21.15.104:3300,v1:172.21.15.104:6789] /tmp/monmap 2024-02-12T16:28:19.085 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-12T16:28:19.085 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:28:19.085 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-12T16:28:19.724 INFO:teuthology.orchestra.run.smithi104.stderr:monmaptool for smithi104 [v2:172.21.15.104:3300,v1:172.21.15.104:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-12T16:28:19.724 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/monmaptool: set fsid to aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:28:19.724 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-12T16:28:19.725 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-02-12T16:28:19.725 INFO:teuthology.orchestra.run.smithi104.stderr:Creating mon... 2024-02-12T16:28:19.725 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/mon.smithi104:/var/lib/ceph/mon/ceph-smithi104:z -v /tmp/ceph-tmpedvxl0ad:/tmp/keyring:z -v /tmp/ceph-tmpntmks2hq:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi104 --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-12T16:28:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.167+0000 7f42c044f700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-12T16:28:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.167+0000 7f42c044f700 1 imported monmap: 2024-02-12T16:28:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-12T16:28:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:28:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: last_changed 2024-02-12T16:28:18.956483+0000 2024-02-12T16:28:20.276 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: created 2024-02-12T16:28:18.956483+0000 2024-02-12T16:28:20.276 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-12T16:28:20.276 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-12T16:28:20.276 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.smithi104 2024-02-12T16:28:20.276 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.276 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.167+0000 7f42c044f700 0 /usr/bin/ceph-mon: set fsid to aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:28:20.276 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: RocksDB version: 6.8.1 2024-02-12T16:28:20.276 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.276 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Compile date May 13 2021 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: DB SUMMARY 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi104/store.db dir, Total Num: 0, files: 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi104/store.db: 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.error_if_exists: 0 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.create_if_missing: 1 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.paranoid_checks: 1 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.env: 0x55a2af38d1c0 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.fs: Posix File System 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.info_log: 0x55a2b08f4ec0 2024-02-12T16:28:20.277 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.statistics: (nil) 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.use_fsync: 0 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.max_log_file_size: 0 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.allow_fallocate: 1 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.use_direct_reads: 0 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-12T16:28:20.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.db_log_dir: 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi104/store.db 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.max_subcompactions: 1 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.max_background_flushes: -1 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-12T16:28:20.279 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.write_buffer_manager: 0x55a2b08fb530 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.unordered_write: 0 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-12T16:28:20.280 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.row_cache: None 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.wal_filter: None 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.preserve_deletes: 0 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.two_write_queues: 0 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.atomic_flush: 0 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.log_readahead_size: 0 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-12T16:28:20.281 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.max_background_jobs: 2 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.max_background_compactions: -1 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.max_open_files: -1 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-12T16:28:20.282 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Compression algorithms supported: 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: kZSTD supported: 0 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: kXpressCompression supported: 0 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: kLZ4Compression supported: 1 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: kBZip2Compression supported: 0 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: kZlibCompression supported: 1 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: kSnappyCompression supported: 1 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.168+0000 7f42c044f700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000001 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.283 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.merge_operator: 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_filter: None 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a2b080dcb8) 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-12T16:28:20.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: block_cache: 0x55a2b0842d10 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-12T16:28:20.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-12T16:28:20.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compression: NoCompression 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.num_levels: 7 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-12T16:28:20.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-12T16:28:20.288 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-12T16:28:20.289 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-12T16:28:20.290 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.table_properties_collectors: 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.inplace_update_support: 0 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.bloom_locality: 0 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.max_successive_merges: 0 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.ttl: 2592000 2024-02-12T16:28:20.291 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.169+0000 7f42c044f700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.170+0000 7f42c044f700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.170+0000 7f42c044f700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.170+0000 7f42c044f700 4 rocksdb: DB pointer 0x55a2b0907800 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.170+0000 7f42a92e8700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.170+0000 7f42a92e8700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:28:20.292 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:28:20.293 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:28:20.294 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:28:20.295 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:28:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:28:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: 2024-02-12T16:28:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.172+0000 7f42c044f700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-12T16:28:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.172+0000 7f42c044f700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-12T16:28:20.296 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-mon: debug 2024-02-12T16:28:20.172+0000 7f42c044f700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi104 for mon.smithi104 2024-02-12T16:28:20.860 INFO:teuthology.orchestra.run.smithi104.stderr:create mon.smithi104 on 2024-02-12T16:28:20.860 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:28:20.866 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl disable ceph.target 2024-02-12T16:28:20.874 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-12T16:28:20.960 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl enable ceph.target 2024-02-12T16:28:20.967 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-12T16:28:21.049 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl start ceph.target 2024-02-12T16:28:21.056 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl enable ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715.target 2024-02-12T16:28:21.062 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715.target. 2024-02-12T16:28:21.062 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715.target. 2024-02-12T16:28:21.139 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl start ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715.target 2024-02-12T16:28:21.147 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl daemon-reload 2024-02-12T16:28:21.235 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl stop ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104 2024-02-12T16:28:21.257 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl reset-failed ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104 2024-02-12T16:28:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Failed to reset failed state of unit ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104.service: Unit ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104.service not loaded. 2024-02-12T16:28:21.262 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl enable ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104 2024-02-12T16:28:21.269 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Created symlink /etc/systemd/system/ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715.target.wants/ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104.service → /etc/systemd/system/ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@.service. 2024-02-12T16:28:21.352 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl start ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104 2024-02-12T16:28:21.918 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-12T16:28:21.925 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: disabled 2024-02-12T16:28:21.926 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-active firewalld.service 2024-02-12T16:28:21.933 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-02-12T16:28:21.933 INFO:teuthology.orchestra.run.smithi104.stderr:firewalld.service is not enabled 2024-02-12T16:28:21.933 INFO:teuthology.orchestra.run.smithi104.stderr:Not possible to enable service . firewalld.service is not available 2024-02-12T16:28:21.934 INFO:teuthology.orchestra.run.smithi104.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-12T16:28:21.934 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mon to start... 2024-02-12T16:28:21.934 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mon... 2024-02-12T16:28:21.934 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/mon.smithi104:/var/lib/ceph/mon/ceph-smithi104:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-12T16:28:22.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:21 smithi104 bash[38986]: 87d81e31db74eb1873e764fac31fed578210bc28517c8fca1132c1c1a218cdc0 2024-02-12T16:28:22.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:21 smithi104 systemd[1]: Started Ceph mon.smithi104 for aa0fd796-c9c3-11ee-95b9-87774f69a715. 2024-02-12T16:28:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.473+0000 7f0f6263e700 1 Processor -- start 2024-02-12T16:28:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.473+0000 7f0f6263e700 1 -- start start 2024-02-12T16:28:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.474+0000 7f0f6263e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.474+0000 7f0f6263e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c0fb6a0 con 0x7f0f5c0fad90 2024-02-12T16:28:22.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.474+0000 7f0f5bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.474+0000 7f0f5bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58714/0 (socket says 172.21.15.104:58714) 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.474+0000 7f0f5bfff700 1 -- 172.21.15.104:0/3042656921 learned_addr learned my addr 172.21.15.104:0/3042656921 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.474+0000 7f0f5bfff700 1 -- 172.21.15.104:0/3042656921 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c0fb7e0 con 0x7f0f5c0fad90 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.474+0000 7f0f5bfff700 1 --2- 172.21.15.104:0/3042656921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c0fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0f4c00b870 tx=0x7f0f4c006f70).ready entity=mon.0 client_cookie=90cac8721f32a704 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.475+0000 7f0f5affd700 1 -- 172.21.15.104:0/3042656921 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f4c00da20 con 0x7f0f5c0fad90 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.475+0000 7f0f5affd700 1 -- 172.21.15.104:0/3042656921 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4c00db80 con 0x7f0f5c0fad90 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.475+0000 7f0f5affd700 1 -- 172.21.15.104:0/3042656921 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f4c00de50 con 0x7f0f5c0fad90 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.475+0000 7f0f6263e700 1 -- 172.21.15.104:0/3042656921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 msgr2=0x7f0f5c0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.475+0000 7f0f6263e700 1 --2- 172.21.15.104:0/3042656921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c0fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0f4c00b870 tx=0x7f0f4c006f70).stop 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.475+0000 7f0f6263e700 1 -- 172.21.15.104:0/3042656921 shutdown_connections 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.475+0000 7f0f6263e700 1 --2- 172.21.15.104:0/3042656921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c0fb1b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.475+0000 7f0f6263e700 1 -- 172.21.15.104:0/3042656921 >> 172.21.15.104:0/3042656921 conn(0x7f0f5c0f6530 msgr2=0x7f0f5c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.475+0000 7f0f6263e700 1 -- 172.21.15.104:0/3042656921 shutdown_connections 2024-02-12T16:28:22.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.475+0000 7f0f6263e700 1 -- 172.21.15.104:0/3042656921 wait complete. 2024-02-12T16:28:22.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.476+0000 7f0f6263e700 1 Processor -- start 2024-02-12T16:28:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.476+0000 7f0f6263e700 1 -- start start 2024-02-12T16:28:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.476+0000 7f0f6263e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c10c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.476+0000 7f0f6263e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c10ca30 con 0x7f0f5c0fad90 2024-02-12T16:28:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f5bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c10c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f5bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c10c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58726/0 (socket says 172.21.15.104:58726) 2024-02-12T16:28:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f5bfff700 1 -- 172.21.15.104:0/1786524958 learned_addr learned my addr 172.21.15.104:0/1786524958 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f5bfff700 1 -- 172.21.15.104:0/1786524958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c00b480 con 0x7f0f5c0fad90 2024-02-12T16:28:22.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f5bfff700 1 --2- 172.21.15.104:0/1786524958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c10c540 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0f4c006c70 tx=0x7f0f4c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f58ff9700 1 -- 172.21.15.104:0/1786524958 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f4c019450 con 0x7f0f5c0fad90 2024-02-12T16:28:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f58ff9700 1 -- 172.21.15.104:0/1786524958 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4c0195b0 con 0x7f0f5c0fad90 2024-02-12T16:28:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f58ff9700 1 -- 172.21.15.104:0/1786524958 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f4c019880 con 0x7f0f5c0fad90 2024-02-12T16:28:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f6263e700 1 -- 172.21.15.104:0/1786524958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c10cc30 con 0x7f0f5c0fad90 2024-02-12T16:28:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f6263e700 1 -- 172.21.15.104:0/1786524958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c10d0c0 con 0x7f0f5c0fad90 2024-02-12T16:28:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f58ff9700 1 -- 172.21.15.104:0/1786524958 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f0f4c02a070 con 0x7f0f5c0fad90 2024-02-12T16:28:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.477+0000 7f0f58ff9700 1 -- 172.21.15.104:0/1786524958 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0f4c0239b0 con 0x7f0f5c0fad90 2024-02-12T16:28:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.478+0000 7f0f6263e700 1 -- 172.21.15.104:0/1786524958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5c045ad0 con 0x7f0f5c0fad90 2024-02-12T16:28:22.482 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.480+0000 7f0f58ff9700 1 -- 172.21.15.104:0/1786524958 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0f4c0429c0 con 0x7f0f5c0fad90 2024-02-12T16:28:22.551 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.546+0000 7f0f6263e700 1 -- 172.21.15.104:0/1786524958 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f0f5c105f90 con 0x7f0f5c0fad90 2024-02-12T16:28:22.551 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.547+0000 7f0f58ff9700 1 -- 172.21.15.104:0/1786524958 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+324 (secure 0 0 0) 0x7f0f4c031070 con 0x7f0f5c0fad90 2024-02-12T16:28:22.551 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: cluster: 2024-02-12T16:28:22.551 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: id: aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:28:22.551 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-12T16:28:22.551 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.551 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: services: 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi104 (age 0.67274s) 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: data: 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: pgs: 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.547+0000 7f0f6263e700 1 -- 172.21.15.104:0/1786524958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 msgr2=0x7f0f5c10c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.547+0000 7f0f6263e700 1 --2- 172.21.15.104:0/1786524958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c10c540 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0f4c006c70 tx=0x7f0f4c006f70).stop 2024-02-12T16:28:22.552 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.547+0000 7f0f6263e700 1 -- 172.21.15.104:0/1786524958 shutdown_connections 2024-02-12T16:28:22.553 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.547+0000 7f0f6263e700 1 --2- 172.21.15.104:0/1786524958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f5c0fad90 0x7f0f5c10c540 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:22.553 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.547+0000 7f0f6263e700 1 -- 172.21.15.104:0/1786524958 >> 172.21.15.104:0/1786524958 conn(0x7f0f5c0f6530 msgr2=0x7f0f5c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:22.553 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.547+0000 7f0f6263e700 1 -- 172.21.15.104:0/1786524958 shutdown_connections 2024-02-12T16:28:22.553 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:22.548+0000 7f0f6263e700 1 -- 172.21.15.104:0/1786524958 wait complete. 2024-02-12T16:28:23.106 INFO:teuthology.orchestra.run.smithi104.stderr:mon is available 2024-02-12T16:28:23.106 INFO:teuthology.orchestra.run.smithi104.stderr:Assimilating anything we can from ceph.conf... 2024-02-12T16:28:23.106 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z -v /var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/mon.smithi104:/var/lib/ceph/mon/ceph-smithi104:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi104/config 2024-02-12T16:28:23.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39001]: cluster 2024-02-12T16:28:21.875490+0000 mon.smithi104 (mon.0) 2 : cluster [INF] mon.smithi104 is new leader, mons smithi104 in quorum (ranks 0) 2024-02-12T16:28:23.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39001]: cluster 2024-02-12T16:28:21.875672+0000 mon.smithi104 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi104=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} 2024-02-12T16:28:23.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39001]: cluster 2024-02-12T16:28:21.878873+0000 mon.smithi104 (mon.0) 4 : cluster [DBG] fsmap 2024-02-12T16:28:23.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39001]: cluster 2024-02-12T16:28:21.882235+0000 mon.smithi104 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-12T16:28:23.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39001]: cluster 2024-02-12T16:28:21.882609+0000 mon.smithi104 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-12T16:28:23.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39001]: audit 2024-02-12T16:28:22.548201+0000 mon.smithi104 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.104:0/1786524958' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-12T16:28:23.687 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.682+0000 7fa0e4b71700 1 Processor -- start 2024-02-12T16:28:23.687 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.682+0000 7fa0e4b71700 1 -- start start 2024-02-12T16:28:23.687 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.682+0000 7fa0e4b71700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e00fe970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:23.687 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.682+0000 7fa0e4b71700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e0070770 con 0x7fa0e0070350 2024-02-12T16:28:23.687 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.682+0000 7fa0df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e00fe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:23.687 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.682+0000 7fa0df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e00fe970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58728/0 (socket says 172.21.15.104:58728) 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.682+0000 7fa0df7fe700 1 -- 172.21.15.104:0/1115560035 learned_addr learned my addr 172.21.15.104:0/1115560035 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.683+0000 7fa0df7fe700 1 -- 172.21.15.104:0/1115560035 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e00fee60 con 0x7fa0e0070350 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.683+0000 7fa0df7fe700 1 --2- 172.21.15.104:0/1115560035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e00fe970 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fa0c800b5f0 tx=0x7fa0c8006ee0).ready entity=mon.0 client_cookie=8aff1a71b848136d server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.683+0000 7fa0de7fc700 1 -- 172.21.15.104:0/1115560035 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c800d600 con 0x7fa0e0070350 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.683+0000 7fa0de7fc700 1 -- 172.21.15.104:0/1115560035 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c800d760 con 0x7fa0e0070350 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.683+0000 7fa0de7fc700 1 -- 172.21.15.104:0/1115560035 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c800da30 con 0x7fa0e0070350 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.683+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1115560035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 msgr2=0x7fa0e00fe970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.683+0000 7fa0e4b71700 1 --2- 172.21.15.104:0/1115560035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e00fe970 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fa0c800b5f0 tx=0x7fa0c8006ee0).stop 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.684+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1115560035 shutdown_connections 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.684+0000 7fa0e4b71700 1 --2- 172.21.15.104:0/1115560035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e00fe970 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.684+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1115560035 >> 172.21.15.104:0/1115560035 conn(0x7fa0e00f6390 msgr2=0x7fa0e00f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.684+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1115560035 shutdown_connections 2024-02-12T16:28:23.688 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.684+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1115560035 wait complete. 2024-02-12T16:28:23.689 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.684+0000 7fa0e4b71700 1 Processor -- start 2024-02-12T16:28:23.689 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.684+0000 7fa0e4b71700 1 -- start start 2024-02-12T16:28:23.689 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.684+0000 7fa0e4b71700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e010a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:23.689 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.684+0000 7fa0e4b71700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e010a6d0 con 0x7fa0e0070350 2024-02-12T16:28:23.689 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e010a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:23.689 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0df7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e010a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58736/0 (socket says 172.21.15.104:58736) 2024-02-12T16:28:23.689 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0df7fe700 1 -- 172.21.15.104:0/1193209531 learned_addr learned my addr 172.21.15.104:0/1193209531 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:23.689 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0df7fe700 1 -- 172.21.15.104:0/1193209531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c800b200 con 0x7fa0e0070350 2024-02-12T16:28:23.689 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0df7fe700 1 --2- 172.21.15.104:0/1193209531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e010a1e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fa0c8006c70 tx=0x7fa0c8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:23.690 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0d7fff700 1 -- 172.21.15.104:0/1193209531 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c8011b40 con 0x7fa0e0070350 2024-02-12T16:28:23.690 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0d7fff700 1 -- 172.21.15.104:0/1193209531 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c8011ca0 con 0x7fa0e0070350 2024-02-12T16:28:23.690 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0d7fff700 1 -- 172.21.15.104:0/1193209531 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c801d430 con 0x7fa0e0070350 2024-02-12T16:28:23.690 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1193209531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e010a8d0 con 0x7fa0e0070350 2024-02-12T16:28:23.690 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1193209531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e010ad60 con 0x7fa0e0070350 2024-02-12T16:28:23.690 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0d7fff700 1 -- 172.21.15.104:0/1193209531 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fa0c800f070 con 0x7fa0e0070350 2024-02-12T16:28:23.690 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.685+0000 7fa0d7fff700 1 -- 172.21.15.104:0/1193209531 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa0c8027460 con 0x7fa0e0070350 2024-02-12T16:28:23.690 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.686+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1193209531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e0045ad0 con 0x7fa0e0070350 2024-02-12T16:28:23.691 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.688+0000 7fa0d7fff700 1 -- 172.21.15.104:0/1193209531 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fa0c8016070 con 0x7fa0e0070350 2024-02-12T16:28:23.754 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.752+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1193209531 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa0e0103ec0 con 0x7fa0e0070350 2024-02-12T16:28:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.755+0000 7fa0d7fff700 1 -- 172.21.15.104:0/1193209531 <== mon.0 v2:172.21.15.104:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa0c801d5e0 con 0x7fa0e0070350 2024-02-12T16:28:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.755+0000 7fa0d7fff700 1 -- 172.21.15.104:0/1193209531 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+491 (secure 0 0 0) 0x7fa0c8036020 con 0x7fa0e0070350 2024-02-12T16:28:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: [global] 2024-02-12T16:28:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: fsid = aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:28:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-12T16:28:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.104:3300,v1:172.21.15.104:6789] 2024-02-12T16:28:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-12T16:28:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-12T16:28:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-12T16:28:23.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: [mgr] 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: [osd] 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.756+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1193209531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 msgr2=0x7fa0e010a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.756+0000 7fa0e4b71700 1 --2- 172.21.15.104:0/1193209531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e010a1e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fa0c8006c70 tx=0x7fa0c8016000).stop 2024-02-12T16:28:23.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.756+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1193209531 shutdown_connections 2024-02-12T16:28:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.756+0000 7fa0e4b71700 1 --2- 172.21.15.104:0/1193209531 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0e0070350 0x7fa0e010a1e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.756+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1193209531 >> 172.21.15.104:0/1193209531 conn(0x7fa0e00f6390 msgr2=0x7fa0e00f8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.756+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1193209531 shutdown_connections 2024-02-12T16:28:23.762 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:23.756+0000 7fa0e4b71700 1 -- 172.21.15.104:0/1193209531 wait complete. 2024-02-12T16:28:24.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:23 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39001]: audit 2024-02-12T16:28:23.753632+0000 mon.smithi104 (mon.0) 8 : audit [INF] from='client.? 172.21.15.104:0/1193209531' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-12T16:28:24.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:23 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39001]: audit 2024-02-12T16:28:23.756482+0000 mon.smithi104 (mon.0) 9 : audit [INF] from='client.? 172.21.15.104:0/1193209531' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-12T16:28:24.354 INFO:teuthology.orchestra.run.smithi104.stderr:Generating new minimal ceph.conf... 2024-02-12T16:28:24.355 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z -v /var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/mon.smithi104:/var/lib/ceph/mon/ceph-smithi104:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi104/config 2024-02-12T16:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.858+0000 7f90a7fff700 1 Processor -- start 2024-02-12T16:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.858+0000 7f90a7fff700 1 -- start start 2024-02-12T16:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.858+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a80fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.858+0000 7f90a7fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a80fed80 con 0x7f90a8070430 2024-02-12T16:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.858+0000 7f90a6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a80fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:24.863 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.858+0000 7f90a6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a80fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58752/0 (socket says 172.21.15.104:58752) 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.858+0000 7f90a6ffd700 1 -- 172.21.15.104:0/783868101 learned_addr learned my addr 172.21.15.104:0/783868101 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.859+0000 7f90a6ffd700 1 -- 172.21.15.104:0/783868101 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a80feec0 con 0x7f90a8070430 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.859+0000 7f90a6ffd700 1 --2- 172.21.15.104:0/783868101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a80fe890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f909800b5f0 tx=0x7f9098006ee0).ready entity=mon.0 client_cookie=fa2457db677a8aab server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.859+0000 7f90a5ffb700 1 -- 172.21.15.104:0/783868101 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f909800d600 con 0x7f90a8070430 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.859+0000 7f90a5ffb700 1 -- 172.21.15.104:0/783868101 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f909800dbf0 con 0x7f90a8070430 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.859+0000 7f90a5ffb700 1 -- 172.21.15.104:0/783868101 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9098011cb0 con 0x7f90a8070430 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.859+0000 7f90a7fff700 1 -- 172.21.15.104:0/783868101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 msgr2=0x7f90a80fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.859+0000 7f90a7fff700 1 --2- 172.21.15.104:0/783868101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a80fe890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f909800b5f0 tx=0x7f9098006ee0).stop 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.859+0000 7f90a7fff700 1 -- 172.21.15.104:0/783868101 shutdown_connections 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.859+0000 7f90a7fff700 1 --2- 172.21.15.104:0/783868101 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a80fe890 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.859+0000 7f90a7fff700 1 -- 172.21.15.104:0/783868101 >> 172.21.15.104:0/783868101 conn(0x7f90a80f6390 msgr2=0x7f90a80f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.860+0000 7f90a7fff700 1 -- 172.21.15.104:0/783868101 shutdown_connections 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.860+0000 7f90a7fff700 1 -- 172.21.15.104:0/783868101 wait complete. 2024-02-12T16:28:24.864 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.860+0000 7f90a7fff700 1 Processor -- start 2024-02-12T16:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.860+0000 7f90a7fff700 1 -- start start 2024-02-12T16:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a810a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f90a7fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a810aec0 con 0x7f90a8070430 2024-02-12T16:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f90a6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a810a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f90a6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a810a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58756/0 (socket says 172.21.15.104:58756) 2024-02-12T16:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f90a6ffd700 1 -- 172.21.15.104:0/3083870995 learned_addr learned my addr 172.21.15.104:0/3083870995 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f90a6ffd700 1 -- 172.21.15.104:0/3083870995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909800b200 con 0x7f90a8070430 2024-02-12T16:28:24.865 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f90a6ffd700 1 --2- 172.21.15.104:0/3083870995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a810a9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f909800df90 tx=0x7f9098006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f908f7fe700 1 -- 172.21.15.104:0/3083870995 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f909801b410 con 0x7f90a8070430 2024-02-12T16:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f908f7fe700 1 -- 172.21.15.104:0/3083870995 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f909801ba00 con 0x7f90a8070430 2024-02-12T16:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f90a7fff700 1 -- 172.21.15.104:0/3083870995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a810b0c0 con 0x7f90a8070430 2024-02-12T16:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f908f7fe700 1 -- 172.21.15.104:0/3083870995 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f909801aa30 con 0x7f90a8070430 2024-02-12T16:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.861+0000 7f90a7fff700 1 -- 172.21.15.104:0/3083870995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a810b550 con 0x7f90a8070430 2024-02-12T16:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.862+0000 7f908f7fe700 1 -- 172.21.15.104:0/3083870995 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f909800f070 con 0x7f90a8070430 2024-02-12T16:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.862+0000 7f908f7fe700 1 -- 172.21.15.104:0/3083870995 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9098025c30 con 0x7f90a8070430 2024-02-12T16:28:24.866 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.863+0000 7f90a7fff700 1 -- 172.21.15.104:0/3083870995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a8045ad0 con 0x7f90a8070430 2024-02-12T16:28:24.867 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.865+0000 7f908f7fe700 1 -- 172.21.15.104:0/3083870995 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9098016070 con 0x7f90a8070430 2024-02-12T16:28:24.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.929+0000 7f90a7fff700 1 -- 172.21.15.104:0/3083870995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f90a80ffc00 con 0x7f90a8070430 2024-02-12T16:28:24.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.929+0000 7f908f7fe700 1 -- 172.21.15.104:0/3083870995 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f9098025e40 con 0x7f90a8070430 2024-02-12T16:28:24.932 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.930+0000 7f90a7fff700 1 -- 172.21.15.104:0/3083870995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 msgr2=0x7f90a810a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:24.932 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.930+0000 7f90a7fff700 1 --2- 172.21.15.104:0/3083870995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a810a9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f909800df90 tx=0x7f9098006ee0).stop 2024-02-12T16:28:24.932 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.930+0000 7f90a7fff700 1 -- 172.21.15.104:0/3083870995 shutdown_connections 2024-02-12T16:28:24.932 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.930+0000 7f90a7fff700 1 --2- 172.21.15.104:0/3083870995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90a8070430 0x7f90a810a9d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:24.933 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.930+0000 7f90a7fff700 1 -- 172.21.15.104:0/3083870995 >> 172.21.15.104:0/3083870995 conn(0x7f90a80f6390 msgr2=0x7f90a80f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:24.933 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.930+0000 7f90a7fff700 1 -- 172.21.15.104:0/3083870995 shutdown_connections 2024-02-12T16:28:24.933 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:24.930+0000 7f90a7fff700 1 -- 172.21.15.104:0/3083870995 wait complete. 2024-02-12T16:28:25.316 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39001]: audit 2024-02-12T16:28:24.930343+0000 mon.smithi104 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.104:0/3083870995' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:28:25.561 INFO:teuthology.orchestra.run.smithi104.stderr:Restarting the monitor... 2024-02-12T16:28:25.561 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl restart ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104 2024-02-12T16:28:25.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:25 smithi104 systemd[1]: Stopping Ceph mon.smithi104 for aa0fd796-c9c3-11ee-95b9-87774f69a715... 2024-02-12T16:28:25.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39001]: debug 2024-02-12T16:28:25.662+0000 7f22d4dc8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi104 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-12T16:28:25.926 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39001]: debug 2024-02-12T16:28:25.662+0000 7f22d4dc8700 -1 mon.smithi104@0(leader) e1 *** Got Signal Terminated *** 2024-02-12T16:28:26.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:26 smithi104 podman[39337]: ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104 2024-02-12T16:28:26.880 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:26 smithi104 systemd[1]: ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104.service: Succeeded. 2024-02-12T16:28:26.880 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:26 smithi104 systemd[1]: Stopped Ceph mon.smithi104 for aa0fd796-c9c3-11ee-95b9-87774f69a715. 2024-02-12T16:28:26.880 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:26 smithi104 systemd[1]: Starting Ceph mon.smithi104 for aa0fd796-c9c3-11ee-95b9-87774f69a715... 2024-02-12T16:28:27.076 INFO:teuthology.orchestra.run.smithi104.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-12T16:28:27.077 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-12T16:28:27.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.010+0000 7ffbff2cb700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-12T16:28:27.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.010+0000 7ffbff2cb700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-12T16:28:27.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.010+0000 7ffbff2cb700 0 pidfile_write: ignore empty --pid-file 2024-02-12T16:28:27.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.014+0000 7ffbff2cb700 0 load: jerasure load: lrc load: isa 2024-02-12T16:28:27.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: RocksDB version: 6.8.1 2024-02-12T16:28:27.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-12T16:28:27.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Compile date May 13 2021 2024-02-12T16:28:27.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: DB SUMMARY 2024-02-12T16:28:27.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: CURRENT file: CURRENT 2024-02-12T16:28:27.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-12T16:28:27.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-12T16:28:27.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi104/store.db dir, Total Num: 1, files: 000004.sst 2024-02-12T16:28:27.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi104/store.db: 000006.log size: 98742 ; 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.error_if_exists: 0 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.create_if_missing: 0 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.paranoid_checks: 1 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.env: 0x559c43ee91c0 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.fs: Posix File System 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.info_log: 0x559c462a9960 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.statistics: (nil) 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.use_fsync: 0 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_log_file_size: 0 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-12T16:28:27.286 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.allow_fallocate: 1 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.use_direct_reads: 0 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.db_log_dir: 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi104/store.db 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_subcompactions: 1 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_background_flushes: -1 2024-02-12T16:28:27.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.write_buffer_manager: 0x559c462affb0 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-12T16:28:27.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.unordered_write: 0 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.row_cache: None 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.wal_filter: None 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-12T16:28:27.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.preserve_deletes: 0 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.two_write_queues: 0 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.atomic_flush: 0 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.log_readahead_size: 0 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_background_jobs: 2 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_background_compactions: -1 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-12T16:28:27.290 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_open_files: -1 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Compression algorithms supported: 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-12T16:28:27.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: kZSTD supported: 0 2024-02-12T16:28:27.292 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: kXpressCompression supported: 0 2024-02-12T16:28:27.292 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-12T16:28:27.292 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: kLZ4Compression supported: 1 2024-02-12T16:28:27.292 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: kBZip2Compression supported: 0 2024-02-12T16:28:27.292 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: kZlibCompression supported: 1 2024-02-12T16:28:27.292 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: kSnappyCompression supported: 1 2024-02-12T16:28:27.292 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-12T16:28:27.292 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000005 2024-02-12T16:28:27.292 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.292 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-12T16:28:27.292 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-12T16:28:27.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.merge_operator: 2024-02-12T16:28:27.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_filter: None 2024-02-12T16:28:27.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-12T16:28:27.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-12T16:28:27.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-12T16:28:27.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559c461c1cb0) 2024-02-12T16:28:27.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cache_index_and_filter_blocks: 1 2024-02-12T16:28:27.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-12T16:28:27.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-12T16:28:27.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: pin_top_level_index_and_filter: 1 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: index_type: 0 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: data_block_index_type: 0 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: index_shortening: 1 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: data_block_hash_table_util_ratio: 0.750000 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: hash_index_allow_collision: 1 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: checksum: 1 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: no_block_cache: 0 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: block_cache: 0x559c461f6f10 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: block_cache_name: BinnedLRUCache 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: block_cache_options: 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: capacity : 536870912 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: num_shard_bits : 4 2024-02-12T16:28:27.294 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: strict_capacity_limit : 0 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: high_pri_pool_ratio: 0.000 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: block_cache_compressed: (nil) 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: persistent_cache: (nil) 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: block_size: 4096 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: block_size_deviation: 10 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: block_restart_interval: 16 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: index_block_restart_interval: 1 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: metadata_block_size: 4096 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: partition_filters: 0 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: use_delta_encoding: 1 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-12T16:28:27.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: whole_key_filtering: 1 2024-02-12T16:28:27.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: verify_compression: 0 2024-02-12T16:28:27.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: read_amp_bytes_per_bit: 0 2024-02-12T16:28:27.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: format_version: 2 2024-02-12T16:28:27.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: enable_index_compression: 1 2024-02-12T16:28:27.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: block_align: 0 2024-02-12T16:28:27.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-12T16:28:27.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-12T16:28:27.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compression: NoCompression 2024-02-12T16:28:27.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-12T16:28:27.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.num_levels: 7 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-12T16:28:27.297 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-12T16:28:27.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-12T16:28:27.299 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-12T16:28:27.300 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-12T16:28:27.300 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-12T16:28:27.300 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-12T16:28:27.300 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-12T16:28:27.300 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-12T16:28:27.300 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-12T16:28:27.300 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-12T16:28:27.300 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-12T16:28:27.300 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-12T16:28:27.300 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-12T16:28:27.300 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.table_properties_collectors: 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.inplace_update_support: 0 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.bloom_locality: 0 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.max_successive_merges: 0 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-12T16:28:27.301 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.ttl: 2592000 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.015+0000 7ffbff2cb700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.016+0000 7ffbff2cb700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.016+0000 7ffbff2cb700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.016+0000 7ffbff2cb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707755307017554, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.016+0000 7ffbff2cb700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.016+0000 7ffbff2cb700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.017+0000 7ffbff2cb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707755307018431, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94316, "table_properties": {"data_size": 92404, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85435, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707755307, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.017+0000 7ffbff2cb700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.017+0000 7ffbff2cb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707755307018831, "job": 1, "event": "recovery_finished"} 2024-02-12T16:28:27.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.018+0000 7ffbff2cb700 4 rocksdb: DB pointer 0x559c462bb800 2024-02-12T16:28:27.303 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.018+0000 7ffbe60e3700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-12T16:28:27.303 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.018+0000 7ffbe60e3700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-12T16:28:27.303 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ** DB Stats ** 2024-02-12T16:28:27.303 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:28:27.303 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-12T16:28:27.303 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-12T16:28:27.303 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:28:27.303 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-12T16:28:27.303 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-12T16:28:27.303 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T16:28:27.303 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ** Compaction Stats [default] ** 2024-02-12T16:28:27.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:28:27.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:28:27.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: L0 2/0 93.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-02-12T16:28:27.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Sum 2/0 93.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-02-12T16:28:27.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-02-12T16:28:27.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ** Compaction Stats [default] ** 2024-02-12T16:28:27.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:28:27.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:28:27.304 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Cumulative compaction: 0.00 GB write, 31.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Interval compaction: 0.00 GB write, 31.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.305 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ** Compaction Stats [default] ** 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: L0 2/0 93.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Sum 2/0 93.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ** Compaction Stats [default] ** 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T16:28:27.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Cumulative compaction: 0.00 GB write, 31.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.019+0000 7ffbff2cb700 0 starting mon.smithi104 rank 0 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi104 fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.020+0000 7ffbff2cb700 1 mon.smithi104@-1(???) e1 preinit fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:28:27.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.020+0000 7ffbff2cb700 0 mon.smithi104@-1(???).mds e1 new map 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.020+0000 7ffbff2cb700 0 mon.smithi104@-1(???).mds e1 print_map 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: e1 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: legacy client fscid: -1 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: No filesystems configured 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.020+0000 7ffbff2cb700 0 mon.smithi104@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.020+0000 7ffbff2cb700 0 mon.smithi104@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.020+0000 7ffbff2cb700 0 mon.smithi104@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:28:27.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.020+0000 7ffbff2cb700 0 mon.smithi104@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:24.930343+0000 mon.smithi104 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.104:0/3083870995' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.020+0000 7ffbff2cb700 1 mon.smithi104@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta expand map: {default=false} 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta from 'false' to 'false' 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta expanded map: {default=false} 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta expand map: {default=info} 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta from 'info' to 'info' 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta expanded map: {default=info} 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta expand map: {default=daemon} 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta expanded map: {default=daemon} 2024-02-12T16:28:27.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta expand map: {default=debug} 2024-02-12T16:28:27.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta from 'debug' to 'debug' 2024-02-12T16:28:27.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:28:27.021+0000 7ffbff2cb700 20 expand_channel_meta expanded map: {default=debug} 2024-02-12T16:28:27.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 bash[39409]: a1087659caf61dda9ee6d55a7a3737d6d1168e7862a1f979b6e7b22be8407986 2024-02-12T16:28:27.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 systemd[1]: Started Ceph mon.smithi104 for aa0fd796-c9c3-11ee-95b9-87774f69a715. 2024-02-12T16:28:27.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:27.036890+0000 mon.smithi104 (mon.0) 1 : cluster [INF] mon.smithi104 is new leader, mons smithi104 in quorum (ranks 0) 2024-02-12T16:28:27.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:27.036949+0000 mon.smithi104 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi104=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} 2024-02-12T16:28:27.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:27.036984+0000 mon.smithi104 (mon.0) 3 : cluster [DBG] fsmap 2024-02-12T16:28:27.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:27.037009+0000 mon.smithi104 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-12T16:28:27.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:27.037414+0000 mon.smithi104 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-12T16:28:27.586 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.578+0000 7fd34ef95700 1 Processor -- start 2024-02-12T16:28:27.587 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.578+0000 7fd34ef95700 1 -- start start 2024-02-12T16:28:27.587 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.579+0000 7fd34ef95700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd3480fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:27.587 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.579+0000 7fd34ef95700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480fef40 con 0x7fd348070430 2024-02-12T16:28:27.587 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.579+0000 7fd34cd31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd3480fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:27.587 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.579+0000 7fd34cd31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd3480fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58772/0 (socket says 172.21.15.104:58772) 2024-02-12T16:28:27.587 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.579+0000 7fd34cd31700 1 -- 172.21.15.104:0/3193804547 learned_addr learned my addr 172.21.15.104:0/3193804547 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:27.587 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.579+0000 7fd34cd31700 1 -- 172.21.15.104:0/3193804547 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480ff080 con 0x7fd348070430 2024-02-12T16:28:27.587 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.580+0000 7fd34cd31700 1 --2- 172.21.15.104:0/3193804547 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd3480fea50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd33c00b5f0 tx=0x7fd33c006ee0).ready entity=mon.0 client_cookie=4f90c68b95741f09 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:27.587 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.580+0000 7fd3477fe700 1 -- 172.21.15.104:0/3193804547 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd33c00d600 con 0x7fd348070430 2024-02-12T16:28:27.587 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.580+0000 7fd3477fe700 1 -- 172.21.15.104:0/3193804547 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd33c00dbf0 con 0x7fd348070430 2024-02-12T16:28:27.587 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.580+0000 7fd3477fe700 1 -- 172.21.15.104:0/3193804547 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd33c011cb0 con 0x7fd348070430 2024-02-12T16:28:27.588 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.580+0000 7fd34ef95700 1 -- 172.21.15.104:0/3193804547 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 msgr2=0x7fd3480fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:27.588 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.580+0000 7fd34ef95700 1 --2- 172.21.15.104:0/3193804547 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd3480fea50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd33c00b5f0 tx=0x7fd33c006ee0).stop 2024-02-12T16:28:27.588 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.580+0000 7fd34ef95700 1 -- 172.21.15.104:0/3193804547 shutdown_connections 2024-02-12T16:28:27.588 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.580+0000 7fd34ef95700 1 --2- 172.21.15.104:0/3193804547 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd3480fea50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:27.588 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.580+0000 7fd34ef95700 1 -- 172.21.15.104:0/3193804547 >> 172.21.15.104:0/3193804547 conn(0x7fd3480f6530 msgr2=0x7fd3480f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:27.588 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.580+0000 7fd34ef95700 1 -- 172.21.15.104:0/3193804547 shutdown_connections 2024-02-12T16:28:27.588 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.581+0000 7fd34ef95700 1 -- 172.21.15.104:0/3193804547 wait complete. 2024-02-12T16:28:27.588 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.581+0000 7fd34ef95700 1 Processor -- start 2024-02-12T16:28:27.588 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.581+0000 7fd34ef95700 1 -- start start 2024-02-12T16:28:27.588 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.581+0000 7fd34ef95700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd34810ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:27.588 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.581+0000 7fd34ef95700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34810b080 con 0x7fd348070430 2024-02-12T16:28:27.589 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.581+0000 7fd34cd31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd34810ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:27.589 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.581+0000 7fd34cd31700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd34810ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58784/0 (socket says 172.21.15.104:58784) 2024-02-12T16:28:27.589 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.581+0000 7fd34cd31700 1 -- 172.21.15.104:0/1891660906 learned_addr learned my addr 172.21.15.104:0/1891660906 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:27.589 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.582+0000 7fd34cd31700 1 -- 172.21.15.104:0/1891660906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c00b200 con 0x7fd348070430 2024-02-12T16:28:27.589 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.582+0000 7fd34cd31700 1 --2- 172.21.15.104:0/1891660906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd34810ab90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd33c00df90 tx=0x7fd33c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:27.589 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.582+0000 7fd3457fa700 1 -- 172.21.15.104:0/1891660906 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd33c01b410 con 0x7fd348070430 2024-02-12T16:28:27.589 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.582+0000 7fd3457fa700 1 -- 172.21.15.104:0/1891660906 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd33c01ba00 con 0x7fd348070430 2024-02-12T16:28:27.589 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.582+0000 7fd3457fa700 1 -- 172.21.15.104:0/1891660906 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd33c01aa30 con 0x7fd348070430 2024-02-12T16:28:27.589 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.582+0000 7fd34ef95700 1 -- 172.21.15.104:0/1891660906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34810b280 con 0x7fd348070430 2024-02-12T16:28:27.589 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.582+0000 7fd34ef95700 1 -- 172.21.15.104:0/1891660906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34810b710 con 0x7fd348070430 2024-02-12T16:28:27.589 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.582+0000 7fd3457fa700 1 -- 172.21.15.104:0/1891660906 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd33c00f070 con 0x7fd348070430 2024-02-12T16:28:27.590 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.582+0000 7fd3457fa700 1 -- 172.21.15.104:0/1891660906 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd33c025c30 con 0x7fd348070430 2024-02-12T16:28:27.590 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.583+0000 7fd34ef95700 1 -- 172.21.15.104:0/1891660906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd348045ad0 con 0x7fd348070430 2024-02-12T16:28:27.590 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.586+0000 7fd3457fa700 1 -- 172.21.15.104:0/1891660906 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd33c016070 con 0x7fd348070430 2024-02-12T16:28:27.651 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.649+0000 7fd34ef95700 1 -- 172.21.15.104:0/1891660906 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd3480ffdc0 con 0x7fd348070430 2024-02-12T16:28:27.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.651+0000 7fd3457fa700 1 -- 172.21.15.104:0/1891660906 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd33c025460 con 0x7fd348070430 2024-02-12T16:28:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.652+0000 7fd34ef95700 1 -- 172.21.15.104:0/1891660906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 msgr2=0x7fd34810ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.652+0000 7fd34ef95700 1 --2- 172.21.15.104:0/1891660906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd34810ab90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd33c00df90 tx=0x7fd33c006ee0).stop 2024-02-12T16:28:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.652+0000 7fd34ef95700 1 -- 172.21.15.104:0/1891660906 shutdown_connections 2024-02-12T16:28:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.652+0000 7fd34ef95700 1 --2- 172.21.15.104:0/1891660906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd348070430 0x7fd34810ab90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.652+0000 7fd34ef95700 1 -- 172.21.15.104:0/1891660906 >> 172.21.15.104:0/1891660906 conn(0x7fd3480f6530 msgr2=0x7fd3480f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.652+0000 7fd34ef95700 1 -- 172.21.15.104:0/1891660906 shutdown_connections 2024-02-12T16:28:27.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:27.652+0000 7fd34ef95700 1 -- 172.21.15.104:0/1891660906 wait complete. 2024-02-12T16:28:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-12T16:28:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:28:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:Creating mgr... 2024-02-12T16:28:28.307 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying port 9283 ... 2024-02-12T16:28:28.309 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl daemon-reload 2024-02-12T16:28:28.403 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl stop ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mgr.smithi104.ohwbhf 2024-02-12T16:28:28.425 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl reset-failed ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mgr.smithi104.ohwbhf 2024-02-12T16:28:28.431 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Failed to reset failed state of unit ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mgr.smithi104.ohwbhf.service: Unit ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mgr.smithi104.ohwbhf.service not loaded. 2024-02-12T16:28:28.431 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl enable ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mgr.smithi104.ohwbhf 2024-02-12T16:28:28.437 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Created symlink /etc/systemd/system/ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715.target.wants/ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mgr.smithi104.ohwbhf.service → /etc/systemd/system/ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@.service. 2024-02-12T16:28:28.519 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl start ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mgr.smithi104.ohwbhf 2024-02-12T16:28:28.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:27.652402+0000 mon.smithi104 (mon.0) 6 : audit [INF] from='client.? 172.21.15.104:0/1891660906' entity='client.admin' 2024-02-12T16:28:29.152 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-12T16:28:29.161 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: disabled 2024-02-12T16:28:29.161 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-active firewalld.service 2024-02-12T16:28:29.168 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-02-12T16:28:29.169 INFO:teuthology.orchestra.run.smithi104.stderr:firewalld.service is not enabled 2024-02-12T16:28:29.169 INFO:teuthology.orchestra.run.smithi104.stderr:Not possible to enable service . firewalld.service is not available 2024-02-12T16:28:29.169 INFO:teuthology.orchestra.run.smithi104.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-12T16:28:29.169 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-12T16:28:29.176 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: disabled 2024-02-12T16:28:29.177 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-active firewalld.service 2024-02-12T16:28:29.184 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-02-12T16:28:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:firewalld.service is not enabled 2024-02-12T16:28:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-12T16:28:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mgr to start... 2024-02-12T16:28:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mgr... 2024-02-12T16:28:29.185 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-12T16:28:29.856 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.802+0000 7f41a89bf700 1 Processor -- start 2024-02-12T16:28:29.856 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.802+0000 7f41a89bf700 1 -- start start 2024-02-12T16:28:29.856 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.802+0000 7f41a89bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a40fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:29.856 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.802+0000 7f41a89bf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a40fb6a0 con 0x7f41a40fad90 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.803+0000 7f41a259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a40fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.803+0000 7f41a259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a40fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58820/0 (socket says 172.21.15.104:58820) 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.803+0000 7f41a259c700 1 -- 172.21.15.104:0/1861176992 learned_addr learned my addr 172.21.15.104:0/1861176992 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.803+0000 7f41a259c700 1 -- 172.21.15.104:0/1861176992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a40fb7e0 con 0x7f41a40fad90 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.803+0000 7f41a259c700 1 --2- 172.21.15.104:0/1861176992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a40fb1b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f419400b870 tx=0x7f4194006f70).ready entity=mon.0 client_cookie=1e07559d6eeeb87b server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.803+0000 7f41a159a700 1 -- 172.21.15.104:0/1861176992 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f419400da20 con 0x7f41a40fad90 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.803+0000 7f41a159a700 1 -- 172.21.15.104:0/1861176992 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4194010470 con 0x7f41a40fad90 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.803+0000 7f41a159a700 1 -- 172.21.15.104:0/1861176992 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f419400f5b0 con 0x7f41a40fad90 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.804+0000 7f41a89bf700 1 -- 172.21.15.104:0/1861176992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 msgr2=0x7f41a40fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.804+0000 7f41a89bf700 1 --2- 172.21.15.104:0/1861176992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a40fb1b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f419400b870 tx=0x7f4194006f70).stop 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.804+0000 7f41a89bf700 1 -- 172.21.15.104:0/1861176992 shutdown_connections 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.804+0000 7f41a89bf700 1 --2- 172.21.15.104:0/1861176992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a40fb1b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:29.857 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.804+0000 7f41a89bf700 1 -- 172.21.15.104:0/1861176992 >> 172.21.15.104:0/1861176992 conn(0x7f41a40f6530 msgr2=0x7f41a40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.804+0000 7f41a89bf700 1 -- 172.21.15.104:0/1861176992 shutdown_connections 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.804+0000 7f41a89bf700 1 -- 172.21.15.104:0/1861176992 wait complete. 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.804+0000 7f41a89bf700 1 Processor -- start 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f41a89bf700 1 -- start start 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f41a89bf700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a410cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f41a89bf700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a410d270 con 0x7f41a40fad90 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f41a259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a410cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f41a259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a410cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58824/0 (socket says 172.21.15.104:58824) 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f41a259c700 1 -- 172.21.15.104:0/2413059238 learned_addr learned my addr 172.21.15.104:0/2413059238 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f41a259c700 1 -- 172.21.15.104:0/2413059238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419400b480 con 0x7f41a40fad90 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f41a259c700 1 --2- 172.21.15.104:0/2413059238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a410cd80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4194006800 tx=0x7f4194006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f4192ffd700 1 -- 172.21.15.104:0/2413059238 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f419400f870 con 0x7f41a40fad90 2024-02-12T16:28:29.858 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f4192ffd700 1 -- 172.21.15.104:0/2413059238 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f419400fe60 con 0x7f41a40fad90 2024-02-12T16:28:29.859 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f41a89bf700 1 -- 172.21.15.104:0/2413059238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a410d470 con 0x7f41a40fad90 2024-02-12T16:28:29.859 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f4192ffd700 1 -- 172.21.15.104:0/2413059238 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4194019e40 con 0x7f41a40fad90 2024-02-12T16:28:29.859 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f41a89bf700 1 -- 172.21.15.104:0/2413059238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a410e0b0 con 0x7f41a40fad90 2024-02-12T16:28:29.859 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f4192ffd700 1 -- 172.21.15.104:0/2413059238 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f419402a070 con 0x7f41a40fad90 2024-02-12T16:28:29.859 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.805+0000 7f4192ffd700 1 -- 172.21.15.104:0/2413059238 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4194023760 con 0x7f41a40fad90 2024-02-12T16:28:29.859 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.806+0000 7f41a89bf700 1 -- 172.21.15.104:0/2413059238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a410d900 con 0x7f41a40fad90 2024-02-12T16:28:29.859 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.808+0000 7f4192ffd700 1 -- 172.21.15.104:0/2413059238 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4194028060 con 0x7f41a40fad90 2024-02-12T16:28:29.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.877+0000 7f41a89bf700 1 -- 172.21.15.104:0/2413059238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f41a4059100 con 0x7f41a40fad90 2024-02-12T16:28:29.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.877+0000 7f4192ffd700 1 -- 172.21.15.104:0/2413059238 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f4194014070 con 0x7f41a40fad90 2024-02-12T16:28:29.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-02-12T16:28:29.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsid": "aa0fd796-c9c3-11ee-95b9-87774f69a715", 2024-02-12T16:28:29.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "health": { 2024-02-12T16:28:29.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-12T16:28:29.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "checks": {}, 2024-02-12T16:28:29.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mutes": [] 2024-02-12T16:28:29.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:29.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-12T16:28:29.885 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum": [ 2024-02-12T16:28:29.885 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 0 2024-02-12T16:28:29.885 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-02-12T16:28:29.885 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-12T16:28:29.885 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "smithi104" 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "monmap": { 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osdmap": { 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-12T16:28:29.886 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgmap": { 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsmap": { 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-12T16:28:29.887 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "available": false, 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modules": [ 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "iostat", 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "restful" 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "servicemap": { 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modified": "2024-02-12T16:28:21.877360+0000", 2024-02-12T16:28:29.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:28:29.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:29.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-12T16:28:29.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-02-12T16:28:29.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.878+0000 7f41a89bf700 1 -- 172.21.15.104:0/2413059238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 msgr2=0x7f41a410cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:29.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.878+0000 7f41a89bf700 1 --2- 172.21.15.104:0/2413059238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a410cd80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4194006800 tx=0x7f4194006f70).stop 2024-02-12T16:28:29.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.878+0000 7f41a89bf700 1 -- 172.21.15.104:0/2413059238 shutdown_connections 2024-02-12T16:28:29.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.878+0000 7f41a89bf700 1 --2- 172.21.15.104:0/2413059238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41a40fad90 0x7f41a410cd80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:29.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.878+0000 7f41a89bf700 1 -- 172.21.15.104:0/2413059238 >> 172.21.15.104:0/2413059238 conn(0x7f41a40f6530 msgr2=0x7f41a40ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:29.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.878+0000 7f41a89bf700 1 -- 172.21.15.104:0/2413059238 shutdown_connections 2024-02-12T16:28:29.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:29.878+0000 7f41a89bf700 1 -- 172.21.15.104:0/2413059238 wait complete. 2024-02-12T16:28:30.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:29 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:29.878833+0000 mon.smithi104 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.104:0/2413059238' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-12T16:28:30.575 INFO:teuthology.orchestra.run.smithi104.stderr:mgr not available, waiting (1/15)... 2024-02-12T16:28:32.577 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-12T16:28:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.119+0000 7f0bd0919700 1 Processor -- start 2024-02-12T16:28:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.119+0000 7f0bd0919700 1 -- start start 2024-02-12T16:28:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.119+0000 7f0bd0919700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc100d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.119+0000 7f0bd0919700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc0fc720 con 0x7f0bcc070390 2024-02-12T16:28:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.120+0000 7f0bcb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc100d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.120+0000 7f0bcb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc100d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54952/0 (socket says 172.21.15.104:54952) 2024-02-12T16:28:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.120+0000 7f0bcb7fe700 1 -- 172.21.15.104:0/918514074 learned_addr learned my addr 172.21.15.104:0/918514074 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.120+0000 7f0bcb7fe700 1 -- 172.21.15.104:0/918514074 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc101220 con 0x7f0bcc070390 2024-02-12T16:28:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.120+0000 7f0bcb7fe700 1 --2- 172.21.15.104:0/918514074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc100d30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f0bb400b5f0 tx=0x7f0bb4006ee0).ready entity=mon.0 client_cookie=3fd399f93af5f364 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.120+0000 7f0bca7fc700 1 -- 172.21.15.104:0/918514074 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bb400d600 con 0x7f0bcc070390 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.120+0000 7f0bca7fc700 1 -- 172.21.15.104:0/918514074 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0bb400dbf0 con 0x7f0bcc070390 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.120+0000 7f0bca7fc700 1 -- 172.21.15.104:0/918514074 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bb4011cb0 con 0x7f0bcc070390 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.121+0000 7f0bd0919700 1 -- 172.21.15.104:0/918514074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 msgr2=0x7f0bcc100d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.121+0000 7f0bd0919700 1 --2- 172.21.15.104:0/918514074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc100d30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f0bb400b5f0 tx=0x7f0bb4006ee0).stop 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.121+0000 7f0bd0919700 1 -- 172.21.15.104:0/918514074 shutdown_connections 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.121+0000 7f0bd0919700 1 --2- 172.21.15.104:0/918514074 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc100d30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.121+0000 7f0bd0919700 1 -- 172.21.15.104:0/918514074 >> 172.21.15.104:0/918514074 conn(0x7f0bcc0f6390 msgr2=0x7f0bcc0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.121+0000 7f0bd0919700 1 -- 172.21.15.104:0/918514074 shutdown_connections 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.121+0000 7f0bd0919700 1 -- 172.21.15.104:0/918514074 wait complete. 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.121+0000 7f0bd0919700 1 Processor -- start 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.121+0000 7f0bd0919700 1 -- start start 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bd0919700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc10cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bd0919700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc10d090 con 0x7f0bcc070390 2024-02-12T16:28:33.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bcb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc10cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bcb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc10cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:54958/0 (socket says 172.21.15.104:54958) 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bcb7fe700 1 -- 172.21.15.104:0/1151492202 learned_addr learned my addr 172.21.15.104:0/1151492202 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bcb7fe700 1 -- 172.21.15.104:0/1151492202 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb400b200 con 0x7f0bcc070390 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bcb7fe700 1 --2- 172.21.15.104:0/1151492202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc10cba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0bb400df90 tx=0x7f0bb4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bc3fff700 1 -- 172.21.15.104:0/1151492202 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bb401b410 con 0x7f0bcc070390 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bc3fff700 1 -- 172.21.15.104:0/1151492202 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0bb401ba00 con 0x7f0bcc070390 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bd0919700 1 -- 172.21.15.104:0/1151492202 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc10d290 con 0x7f0bcc070390 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bc3fff700 1 -- 172.21.15.104:0/1151492202 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bb401aa30 con 0x7f0bcc070390 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bd0919700 1 -- 172.21.15.104:0/1151492202 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc10d720 con 0x7f0bcc070390 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.122+0000 7f0bc3fff700 1 -- 172.21.15.104:0/1151492202 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f0bb400f070 con 0x7f0bcc070390 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.123+0000 7f0bc3fff700 1 -- 172.21.15.104:0/1151492202 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0bb4025c80 con 0x7f0bcc070390 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.123+0000 7f0bd0919700 1 -- 172.21.15.104:0/1151492202 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc045ad0 con 0x7f0bcc070390 2024-02-12T16:28:33.139 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.125+0000 7f0bc3fff700 1 -- 172.21.15.104:0/1151492202 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0bb4016070 con 0x7f0bcc070390 2024-02-12T16:28:33.202 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.195+0000 7f0bd0919700 1 -- 172.21.15.104:0/1151492202 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0bcc106cb0 con 0x7f0bcc070390 2024-02-12T16:28:33.202 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.195+0000 7f0bc3fff700 1 -- 172.21.15.104:0/1151492202 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f0bb4025460 con 0x7f0bcc070390 2024-02-12T16:28:33.202 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.202 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-02-12T16:28:33.202 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsid": "aa0fd796-c9c3-11ee-95b9-87774f69a715", 2024-02-12T16:28:33.202 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "health": { 2024-02-12T16:28:33.202 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "checks": {}, 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mutes": [] 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum": [ 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 0 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "smithi104" 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "monmap": { 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:33.203 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osdmap": { 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgmap": { 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-12T16:28:33.204 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsmap": { 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-12T16:28:33.205 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "available": false, 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modules": [ 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "iostat", 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "restful" 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "servicemap": { 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modified": "2024-02-12T16:28:21.877360+0000", 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:33.206 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-12T16:28:33.207 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-02-12T16:28:33.208 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.197+0000 7f0bd0919700 1 -- 172.21.15.104:0/1151492202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 msgr2=0x7f0bcc10cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:33.208 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.197+0000 7f0bd0919700 1 --2- 172.21.15.104:0/1151492202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc10cba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0bb400df90 tx=0x7f0bb4006ee0).stop 2024-02-12T16:28:33.208 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.197+0000 7f0bd0919700 1 -- 172.21.15.104:0/1151492202 shutdown_connections 2024-02-12T16:28:33.208 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.197+0000 7f0bd0919700 1 --2- 172.21.15.104:0/1151492202 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bcc070390 0x7f0bcc10cba0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:33.208 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.197+0000 7f0bd0919700 1 -- 172.21.15.104:0/1151492202 >> 172.21.15.104:0/1151492202 conn(0x7f0bcc0f6390 msgr2=0x7f0bcc0f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:33.208 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.197+0000 7f0bd0919700 1 -- 172.21.15.104:0/1151492202 shutdown_connections 2024-02-12T16:28:33.208 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:33.197+0000 7f0bd0919700 1 -- 172.21.15.104:0/1151492202 wait complete. 2024-02-12T16:28:33.603 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:33.195382+0000 mon.smithi104 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.104:0/1151492202' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-12T16:28:33.813 INFO:teuthology.orchestra.run.smithi104.stderr:mgr not available, waiting (2/15)... 2024-02-12T16:28:34.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:33.920706+0000 mon.smithi104 (mon.0) 9 : cluster [INF] Activating manager daemon smithi104.ohwbhf 2024-02-12T16:28:34.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:33.923126+0000 mon.smithi104 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi104.ohwbhf(active, starting, since 0.00249739s) 2024-02-12T16:28:34.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:33.924034+0000 mon.smithi104 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.104:0/4015962675' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T16:28:34.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:33.924174+0000 mon.smithi104 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.104:0/4015962675' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T16:28:34.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:33.924353+0000 mon.smithi104 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.104:0/4015962675' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T16:28:34.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:33.924532+0000 mon.smithi104 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.104:0/4015962675' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-12T16:28:34.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:33.924793+0000 mon.smithi104 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.104:0/4015962675' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mgr metadata", "who": "smithi104.ohwbhf", "id": "smithi104.ohwbhf"}]: dispatch 2024-02-12T16:28:34.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:33.938472+0000 mon.smithi104 (mon.0) 16 : cluster [INF] Manager daemon smithi104.ohwbhf is now available 2024-02-12T16:28:34.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:33.945182+0000 mon.smithi104 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.104:0/4015962675' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:28:34.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:33.950019+0000 mon.smithi104 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.104:0/4015962675' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:28:34.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:33.951085+0000 mon.smithi104 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.104:0/4015962675' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:28:34.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:33.955604+0000 mon.smithi104 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.104:0/4015962675' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:28:34.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:33.957495+0000 mon.smithi104 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.104:0/4015962675' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:28:35.816 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-12T16:28:36.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:35 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:34.926854+0000 mon.smithi104 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi104.ohwbhf(active, since 1.00622s) 2024-02-12T16:28:36.708 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.705+0000 7f8119a1e700 1 Processor -- start 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.705+0000 7f8119a1e700 1 -- start start 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.705+0000 7f8119a1e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.705+0000 7f8119a1e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81140fc6a0 con 0x7f81140fbd90 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.705+0000 7f8118a1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.705+0000 7f8118a1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55014/0 (socket says 172.21.15.104:55014) 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.705+0000 7f8118a1c700 1 -- 172.21.15.104:0/2597611231 learned_addr learned my addr 172.21.15.104:0/2597611231 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.705+0000 7f8118a1c700 1 -- 172.21.15.104:0/2597611231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81140fc7e0 con 0x7f81140fbd90 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.705+0000 7f8118a1c700 1 --2- 172.21.15.104:0/2597611231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140fc1b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f810800b5f0 tx=0x7f8108006ee0).ready entity=mon.0 client_cookie=a7051a71640471d2 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.706+0000 7f81137fe700 1 -- 172.21.15.104:0/2597611231 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f810800d600 con 0x7f81140fbd90 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.706+0000 7f81137fe700 1 -- 172.21.15.104:0/2597611231 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f810800dbf0 con 0x7f81140fbd90 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.706+0000 7f81137fe700 1 -- 172.21.15.104:0/2597611231 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8108011cb0 con 0x7f81140fbd90 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.706+0000 7f8119a1e700 1 -- 172.21.15.104:0/2597611231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 msgr2=0x7f81140fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:36.709 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.706+0000 7f8119a1e700 1 --2- 172.21.15.104:0/2597611231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140fc1b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f810800b5f0 tx=0x7f8108006ee0).stop 2024-02-12T16:28:36.710 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.706+0000 7f8119a1e700 1 -- 172.21.15.104:0/2597611231 shutdown_connections 2024-02-12T16:28:36.710 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.706+0000 7f8119a1e700 1 --2- 172.21.15.104:0/2597611231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140fc1b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:36.710 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.706+0000 7f8119a1e700 1 -- 172.21.15.104:0/2597611231 >> 172.21.15.104:0/2597611231 conn(0x7f81140f7510 msgr2=0x7f81140f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:36.710 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.707+0000 7f8119a1e700 1 -- 172.21.15.104:0/2597611231 shutdown_connections 2024-02-12T16:28:36.710 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.707+0000 7f8119a1e700 1 -- 172.21.15.104:0/2597611231 wait complete. 2024-02-12T16:28:36.710 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.707+0000 7f8119a1e700 1 Processor -- start 2024-02-12T16:28:36.710 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.707+0000 7f8119a1e700 1 -- start start 2024-02-12T16:28:36.712 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.708+0000 7f8119a1e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:36.712 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.708+0000 7f8119a1e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8114070a60 con 0x7f81140fbd90 2024-02-12T16:28:36.712 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.708+0000 7f8118a1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:36.712 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.708+0000 7f8118a1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55026/0 (socket says 172.21.15.104:55026) 2024-02-12T16:28:36.712 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.708+0000 7f8118a1c700 1 -- 172.21.15.104:0/459756332 learned_addr learned my addr 172.21.15.104:0/459756332 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:36.712 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.708+0000 7f8118a1c700 1 -- 172.21.15.104:0/459756332 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810800b200 con 0x7f81140fbd90 2024-02-12T16:28:36.712 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.708+0000 7f8118a1c700 1 --2- 172.21.15.104:0/459756332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140725e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8108011ea0 tx=0x7f8108006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.708+0000 7f81117fa700 1 -- 172.21.15.104:0/459756332 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f810801b570 con 0x7f81140fbd90 2024-02-12T16:28:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.708+0000 7f8119a1e700 1 -- 172.21.15.104:0/459756332 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8114070c00 con 0x7f81140fbd90 2024-02-12T16:28:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.708+0000 7f81117fa700 1 -- 172.21.15.104:0/459756332 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f810801bb60 con 0x7f81140fbd90 2024-02-12T16:28:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.708+0000 7f81117fa700 1 -- 172.21.15.104:0/459756332 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f810801a8d0 con 0x7f81140fbd90 2024-02-12T16:28:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.709+0000 7f8119a1e700 1 -- 172.21.15.104:0/459756332 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8114071090 con 0x7f81140fbd90 2024-02-12T16:28:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.709+0000 7f81117fa700 1 -- 172.21.15.104:0/459756332 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f810800f070 con 0x7f81140fbd90 2024-02-12T16:28:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.709+0000 7f81117fa700 1 --2- 172.21.15.104:0/459756332 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f80f802fd10 0x7f80f80321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.709+0000 7f81117fa700 1 -- 172.21.15.104:0/459756332 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f81080412c0 con 0x7f81140fbd90 2024-02-12T16:28:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.709+0000 7f8113fff700 1 --2- 172.21.15.104:0/459756332 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f80f802fd10 0x7f80f80321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.710+0000 7f8113fff700 1 --2- 172.21.15.104:0/459756332 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f80f802fd10 0x7f80f80321d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f81000096a0 tx=0x7f8100006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:36.713 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.710+0000 7f8119a1e700 1 -- 172.21.15.104:0/459756332 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8114059100 con 0x7f81140fbd90 2024-02-12T16:28:36.715 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.714+0000 7f81117fa700 1 -- 172.21.15.104:0/459756332 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8108016070 con 0x7f81140fbd90 2024-02-12T16:28:36.926 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.917+0000 7f8119a1e700 1 -- 172.21.15.104:0/459756332 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8114045ad0 con 0x7f81140fbd90 2024-02-12T16:28:36.926 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.917+0000 7f81117fa700 1 -- 172.21.15.104:0/459756332 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f8108022b70 con 0x7f81140fbd90 2024-02-12T16:28:36.926 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.926 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsid": "aa0fd796-c9c3-11ee-95b9-87774f69a715", 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "health": { 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "checks": {}, 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mutes": [] 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum": [ 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 0 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "smithi104" 2024-02-12T16:28:36.927 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "monmap": { 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osdmap": { 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-12T16:28:36.928 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgmap": { 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsmap": { 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-12T16:28:36.929 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "available": true, 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modules": [ 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "iostat", 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "restful" 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "servicemap": { 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modified": "2024-02-12T16:28:21.877360+0000", 2024-02-12T16:28:36.930 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.919+0000 7f8119a1e700 1 -- 172.21.15.104:0/459756332 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f80f802fd10 msgr2=0x7f80f80321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.919+0000 7f8119a1e700 1 --2- 172.21.15.104:0/459756332 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f80f802fd10 0x7f80f80321d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f81000096a0 tx=0x7f8100006b40).stop 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.919+0000 7f8119a1e700 1 -- 172.21.15.104:0/459756332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 msgr2=0x7f81140725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.919+0000 7f8119a1e700 1 --2- 172.21.15.104:0/459756332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140725e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8108011ea0 tx=0x7f8108006ee0).stop 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.919+0000 7f8119a1e700 1 -- 172.21.15.104:0/459756332 shutdown_connections 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.919+0000 7f8119a1e700 1 --2- 172.21.15.104:0/459756332 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f80f802fd10 0x7f80f80321d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.919+0000 7f8119a1e700 1 --2- 172.21.15.104:0/459756332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f81140fbd90 0x7f81140725e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.919+0000 7f8119a1e700 1 -- 172.21.15.104:0/459756332 >> 172.21.15.104:0/459756332 conn(0x7f81140f7510 msgr2=0x7f81140f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.919+0000 7f8119a1e700 1 -- 172.21.15.104:0/459756332 shutdown_connections 2024-02-12T16:28:36.931 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:36.919+0000 7f8119a1e700 1 -- 172.21.15.104:0/459756332 wait complete. 2024-02-12T16:28:37.236 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:36.917888+0000 mon.smithi104 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.104:0/459756332' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-12T16:28:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:mgr is available 2024-02-12T16:28:37.506 INFO:teuthology.orchestra.run.smithi104.stderr:Enabling cephadm module... 2024-02-12T16:28:37.507 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.020+0000 7f1aa9f01700 1 Processor -- start 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.020+0000 7f1aa9f01700 1 -- start start 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.021+0000 7f1aa9f01700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa40fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.021+0000 7f1aa9f01700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa40fef40 con 0x7f1aa4070430 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.021+0000 7f1aa37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa40fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.021+0000 7f1aa37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa40fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55034/0 (socket says 172.21.15.104:55034) 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.021+0000 7f1aa37fe700 1 -- 172.21.15.104:0/3359205079 learned_addr learned my addr 172.21.15.104:0/3359205079 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.021+0000 7f1aa37fe700 1 -- 172.21.15.104:0/3359205079 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa40ff080 con 0x7f1aa4070430 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.021+0000 7f1aa37fe700 1 --2- 172.21.15.104:0/3359205079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa40fea50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1a9800b5f0 tx=0x7f1a98006ee0).ready entity=mon.0 client_cookie=ed6a3ea27ec70823 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.021+0000 7f1aa27fc700 1 -- 172.21.15.104:0/3359205079 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9800d600 con 0x7f1aa4070430 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.021+0000 7f1aa27fc700 1 -- 172.21.15.104:0/3359205079 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1a9800dbf0 con 0x7f1aa4070430 2024-02-12T16:28:38.069 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.021+0000 7f1aa27fc700 1 -- 172.21.15.104:0/3359205079 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a98011cb0 con 0x7f1aa4070430 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.022+0000 7f1aa9f01700 1 -- 172.21.15.104:0/3359205079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 msgr2=0x7f1aa40fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.022+0000 7f1aa9f01700 1 --2- 172.21.15.104:0/3359205079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa40fea50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1a9800b5f0 tx=0x7f1a98006ee0).stop 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.022+0000 7f1aa9f01700 1 -- 172.21.15.104:0/3359205079 shutdown_connections 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.022+0000 7f1aa9f01700 1 --2- 172.21.15.104:0/3359205079 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa40fea50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.022+0000 7f1aa9f01700 1 -- 172.21.15.104:0/3359205079 >> 172.21.15.104:0/3359205079 conn(0x7f1aa40f6530 msgr2=0x7f1aa40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.022+0000 7f1aa9f01700 1 -- 172.21.15.104:0/3359205079 shutdown_connections 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.022+0000 7f1aa9f01700 1 -- 172.21.15.104:0/3359205079 wait complete. 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.022+0000 7f1aa9f01700 1 Processor -- start 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.023+0000 7f1aa9f01700 1 -- start start 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.023+0000 7f1aa9f01700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa410ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.023+0000 7f1aa9f01700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa410b020 con 0x7f1aa4070430 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.023+0000 7f1aa37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa410ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.023+0000 7f1aa37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa410ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55036/0 (socket says 172.21.15.104:55036) 2024-02-12T16:28:38.070 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.023+0000 7f1aa37fe700 1 -- 172.21.15.104:0/818226358 learned_addr learned my addr 172.21.15.104:0/818226358 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.023+0000 7f1aa37fe700 1 -- 172.21.15.104:0/818226358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9800b200 con 0x7f1aa4070430 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.023+0000 7f1aa37fe700 1 --2- 172.21.15.104:0/818226358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa410ab30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1a98006800 tx=0x7f1a98006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.023+0000 7f1a8bfff700 1 -- 172.21.15.104:0/818226358 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9801b570 con 0x7f1aa4070430 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.023+0000 7f1a8bfff700 1 -- 172.21.15.104:0/818226358 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1a9801bb60 con 0x7f1aa4070430 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.024+0000 7f1aa9f01700 1 -- 172.21.15.104:0/818226358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa410b220 con 0x7f1aa4070430 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.024+0000 7f1a8bfff700 1 -- 172.21.15.104:0/818226358 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9801a9f0 con 0x7f1aa4070430 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.024+0000 7f1aa9f01700 1 -- 172.21.15.104:0/818226358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa410b6b0 con 0x7f1aa4070430 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.024+0000 7f1a8bfff700 1 -- 172.21.15.104:0/818226358 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f1a9800f070 con 0x7f1aa4070430 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.024+0000 7f1aa9f01700 1 -- 172.21.15.104:0/818226358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa4045ad0 con 0x7f1aa4070430 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.024+0000 7f1a8bfff700 1 --2- 172.21.15.104:0/818226358 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f1a9002fe10 0x7f1a900322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.024+0000 7f1a8bfff700 1 -- 172.21.15.104:0/818226358 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1a98041280 con 0x7f1aa4070430 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.025+0000 7f1aa2ffd700 1 --2- 172.21.15.104:0/818226358 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f1a9002fe10 0x7f1a900322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.025+0000 7f1aa2ffd700 1 --2- 172.21.15.104:0/818226358 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f1a9002fe10 0x7f1a900322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1a940096a0 tx=0x7f1a94006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:38.071 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.029+0000 7f1a8bfff700 1 -- 172.21.15.104:0/818226358 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a98022780 con 0x7f1aa4070430 2024-02-12T16:28:38.214 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.213+0000 7f1aa9f01700 1 -- 172.21.15.104:0/818226358 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f1aa40ffdc0 con 0x7f1aa4070430 2024-02-12T16:28:38.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:37 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:36.926077+0000 mon.smithi104 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi104.ohwbhf(active, since 3s) 2024-02-12T16:28:38.932 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.931+0000 7f1a8bfff700 1 -- 172.21.15.104:0/818226358 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f1a9801bcd0 con 0x7f1aa4070430 2024-02-12T16:28:38.933 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.932+0000 7f1a8bfff700 1 -- 172.21.15.104:0/818226358 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f1a98019770 con 0x7f1aa4070430 2024-02-12T16:28:38.936 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.934+0000 7f1aa9f01700 1 -- 172.21.15.104:0/818226358 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f1a9002fe10 msgr2=0x7f1a900322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:38.936 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.934+0000 7f1aa9f01700 1 --2- 172.21.15.104:0/818226358 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f1a9002fe10 0x7f1a900322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1a940096a0 tx=0x7f1a94006b40).stop 2024-02-12T16:28:38.936 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.934+0000 7f1aa9f01700 1 -- 172.21.15.104:0/818226358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 msgr2=0x7f1aa410ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:38.936 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.934+0000 7f1aa9f01700 1 --2- 172.21.15.104:0/818226358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa410ab30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1a98006800 tx=0x7f1a98006ee0).stop 2024-02-12T16:28:38.936 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.934+0000 7f1aa9f01700 1 -- 172.21.15.104:0/818226358 shutdown_connections 2024-02-12T16:28:38.936 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.934+0000 7f1aa9f01700 1 --2- 172.21.15.104:0/818226358 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7f1a9002fe10 0x7f1a900322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:38.936 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.934+0000 7f1aa9f01700 1 --2- 172.21.15.104:0/818226358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1aa4070430 0x7f1aa410ab30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:38.937 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.934+0000 7f1aa9f01700 1 -- 172.21.15.104:0/818226358 >> 172.21.15.104:0/818226358 conn(0x7f1aa40f6530 msgr2=0x7f1aa40f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:38.937 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.935+0000 7f1aa9f01700 1 -- 172.21.15.104:0/818226358 shutdown_connections 2024-02-12T16:28:38.937 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:38.935+0000 7f1aa9f01700 1 -- 172.21.15.104:0/818226358 wait complete. 2024-02-12T16:28:39.352 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:38.213738+0000 mon.smithi104 (mon.0) 25 : audit [INF] from='client.? 172.21.15.104:0/818226358' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-12T16:28:39.637 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-12T16:28:40.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:39 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:38.932215+0000 mon.smithi104 (mon.0) 26 : audit [INF] from='client.? 172.21.15.104:0/818226358' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-12T16:28:40.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:39 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:38.932273+0000 mon.smithi104 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi104.ohwbhf(active, since 5s) 2024-02-12T16:28:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.251+0000 7fd767750700 1 Processor -- start 2024-02-12T16:28:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.251+0000 7fd767750700 1 -- start start 2024-02-12T16:28:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.252+0000 7fd767750700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd7600fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.252+0000 7fd767750700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600fb6a0 con 0x7fd7600fad90 2024-02-12T16:28:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.252+0000 7fd7654ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd7600fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.252+0000 7fd7654ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd7600fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55072/0 (socket says 172.21.15.104:55072) 2024-02-12T16:28:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.252+0000 7fd7654ec700 1 -- 172.21.15.104:0/613323488 learned_addr learned my addr 172.21.15.104:0/613323488 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:40.255 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.252+0000 7fd7654ec700 1 -- 172.21.15.104:0/613323488 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7600fb7e0 con 0x7fd7600fad90 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.252+0000 7fd7654ec700 1 --2- 172.21.15.104:0/613323488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd7600fb1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fd75c00b5f0 tx=0x7fd75c006ee0).ready entity=mon.0 client_cookie=a127a31cddef6e3b server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.252+0000 7fd74ffff700 1 -- 172.21.15.104:0/613323488 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd75c00d600 con 0x7fd7600fad90 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.252+0000 7fd74ffff700 1 -- 172.21.15.104:0/613323488 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd75c00dbf0 con 0x7fd7600fad90 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.253+0000 7fd74ffff700 1 -- 172.21.15.104:0/613323488 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd75c011cb0 con 0x7fd7600fad90 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.253+0000 7fd767750700 1 -- 172.21.15.104:0/613323488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 msgr2=0x7fd7600fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.253+0000 7fd767750700 1 --2- 172.21.15.104:0/613323488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd7600fb1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fd75c00b5f0 tx=0x7fd75c006ee0).stop 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.253+0000 7fd767750700 1 -- 172.21.15.104:0/613323488 shutdown_connections 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.253+0000 7fd767750700 1 --2- 172.21.15.104:0/613323488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd7600fb1b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.253+0000 7fd767750700 1 -- 172.21.15.104:0/613323488 >> 172.21.15.104:0/613323488 conn(0x7fd7600f6530 msgr2=0x7fd7600f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.253+0000 7fd767750700 1 -- 172.21.15.104:0/613323488 shutdown_connections 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.253+0000 7fd767750700 1 -- 172.21.15.104:0/613323488 wait complete. 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.254+0000 7fd767750700 1 Processor -- start 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.254+0000 7fd767750700 1 -- start start 2024-02-12T16:28:40.256 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.254+0000 7fd767750700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd76010cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:40.257 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.254+0000 7fd767750700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76010d280 con 0x7fd7600fad90 2024-02-12T16:28:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.254+0000 7fd7654ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd76010cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.254+0000 7fd7654ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd76010cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:55080/0 (socket says 172.21.15.104:55080) 2024-02-12T16:28:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.254+0000 7fd7654ec700 1 -- 172.21.15.104:0/4208152470 learned_addr learned my addr 172.21.15.104:0/4208152470 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.255+0000 7fd7654ec700 1 -- 172.21.15.104:0/4208152470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75c00b200 con 0x7fd7600fad90 2024-02-12T16:28:40.258 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.255+0000 7fd7654ec700 1 --2- 172.21.15.104:0/4208152470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd76010cd90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fd75c011ea0 tx=0x7fd75c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.255+0000 7fd74dffb700 1 -- 172.21.15.104:0/4208152470 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd75c01b570 con 0x7fd7600fad90 2024-02-12T16:28:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.255+0000 7fd74dffb700 1 -- 172.21.15.104:0/4208152470 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd75c01bb60 con 0x7fd7600fad90 2024-02-12T16:28:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.255+0000 7fd767750700 1 -- 172.21.15.104:0/4208152470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76010d480 con 0x7fd7600fad90 2024-02-12T16:28:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.255+0000 7fd74dffb700 1 -- 172.21.15.104:0/4208152470 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd75c01a8d0 con 0x7fd7600fad90 2024-02-12T16:28:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.255+0000 7fd767750700 1 -- 172.21.15.104:0/4208152470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76010e090 con 0x7fd7600fad90 2024-02-12T16:28:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.256+0000 7fd74dffb700 1 -- 172.21.15.104:0/4208152470 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd75c00f070 con 0x7fd7600fad90 2024-02-12T16:28:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.256+0000 7fd74dffb700 1 --2- 172.21.15.104:0/4208152470 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7fd74802fe70 0x7fd748032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.256+0000 7fd74dffb700 1 -- 172.21.15.104:0/4208152470 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd75c042210 con 0x7fd7600fad90 2024-02-12T16:28:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.256+0000 7fd764ceb700 1 -- 172.21.15.104:0/4208152470 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7fd74802fe70 msgr2=0x7fd748032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/4286295317 2024-02-12T16:28:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.256+0000 7fd764ceb700 1 --2- 172.21.15.104:0/4208152470 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7fd74802fe70 0x7fd748032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-12T16:28:40.259 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.256+0000 7fd767750700 1 -- 172.21.15.104:0/4208152470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd760059100 con 0x7fd7600fad90 2024-02-12T16:28:40.262 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.260+0000 7fd74dffb700 1 -- 172.21.15.104:0/4208152470 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd75c016070 con 0x7fd7600fad90 2024-02-12T16:28:40.456 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.453+0000 7fd767750700 1 -- 172.21.15.104:0/4208152470 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd7600fcae0 con 0x7fd7600fad90 2024-02-12T16:28:40.456 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.453+0000 7fd74dffb700 1 -- 172.21.15.104:0/4208152470 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fd75c022b70 con 0x7fd7600fad90 2024-02-12T16:28:40.456 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-02-12T16:28:40.456 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-12T16:28:40.457 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "available": true, 2024-02-12T16:28:40.457 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "active_name": "smithi104.ohwbhf", 2024-02-12T16:28:40.457 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-12T16:28:40.457 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-02-12T16:28:40.458 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.455+0000 7fd767750700 1 -- 172.21.15.104:0/4208152470 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7fd74802fe70 msgr2=0x7fd748032330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-12T16:28:40.458 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.455+0000 7fd767750700 1 --2- 172.21.15.104:0/4208152470 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7fd74802fe70 0x7fd748032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-12T16:28:40.458 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.455+0000 7fd767750700 1 -- 172.21.15.104:0/4208152470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 msgr2=0x7fd76010cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:40.459 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.455+0000 7fd767750700 1 --2- 172.21.15.104:0/4208152470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd76010cd90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fd75c011ea0 tx=0x7fd75c006ee0).stop 2024-02-12T16:28:40.459 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.455+0000 7fd767750700 1 -- 172.21.15.104:0/4208152470 shutdown_connections 2024-02-12T16:28:40.459 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.455+0000 7fd767750700 1 --2- 172.21.15.104:0/4208152470 >> [v2:172.21.15.104:6800/4286295317,v1:172.21.15.104:6801/4286295317] conn(0x7fd74802fe70 0x7fd748032330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-12T16:28:40.459 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.455+0000 7fd767750700 1 --2- 172.21.15.104:0/4208152470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7600fad90 0x7fd76010cd90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:40.459 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.455+0000 7fd767750700 1 -- 172.21.15.104:0/4208152470 >> 172.21.15.104:0/4208152470 conn(0x7fd7600f6530 msgr2=0x7fd7600700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:40.459 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.455+0000 7fd767750700 1 -- 172.21.15.104:0/4208152470 shutdown_connections 2024-02-12T16:28:40.459 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:40.456+0000 7fd767750700 1 -- 172.21.15.104:0/4208152470 wait complete. 2024-02-12T16:28:41.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:40.454787+0000 mon.smithi104 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.104:0/4208152470' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-12T16:28:43.322 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for the mgr to restart... 2024-02-12T16:28:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mgr epoch 5... 2024-02-12T16:28:43.323 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-12T16:28:44.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:43 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:43.841812+0000 mon.smithi104 (mon.0) 29 : cluster [INF] Active manager daemon smithi104.ohwbhf restarted 2024-02-12T16:28:44.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:43 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:43.842176+0000 mon.smithi104 (mon.0) 30 : cluster [INF] Activating manager daemon smithi104.ohwbhf 2024-02-12T16:28:44.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:43 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:43.846329+0000 mon.smithi104 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-12T16:28:45.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:43.894765+0000 mon.smithi104 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi104.ohwbhf(active, starting, since 0.0526676s) 2024-02-12T16:28:45.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:43.896992+0000 mon.smithi104 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-12T16:28:45.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:43.897629+0000 mon.smithi104 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mgr metadata", "who": "smithi104.ohwbhf", "id": "smithi104.ohwbhf"}]: dispatch 2024-02-12T16:28:45.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:43.898264+0000 mon.smithi104 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T16:28:45.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:43.898474+0000 mon.smithi104 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T16:28:45.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:43.898679+0000 mon.smithi104 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T16:28:45.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:43.912366+0000 mon.smithi104 (mon.0) 38 : cluster [INF] Manager daemon smithi104.ohwbhf is now available 2024-02-12T16:28:45.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:43.925960+0000 mon.smithi104 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:28:45.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:43.928627+0000 mon.smithi104 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:28:45.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:43.931028+0000 mon.smithi104 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:28:45.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:44.019022+0000 mon.smithi104 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:28:45.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:44.019752+0000 mon.smithi104 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:28:45.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:44.021414+0000 mon.smithi104 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:28:45.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:44.022810+0000 mon.smithi104 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:28:45.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:44.032292+0000 mon.smithi104 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:28:45.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:44.033400+0000 mon.smithi104 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:28:46.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:45 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:44.909036+0000 mon.smithi104 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi104.ohwbhf(active, since 1.06694s) 2024-02-12T16:28:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.830+0000 7fbaece10700 1 Processor -- start 2024-02-12T16:28:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.830+0000 7fbaece10700 1 -- start start 2024-02-12T16:28:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.830+0000 7fbaece10700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae80fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.830+0000 7fbaece10700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae80fef80 con 0x7fbae80703f0 2024-02-12T16:28:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.830+0000 7fbae659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae80fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.830+0000 7fbae659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae80fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35904/0 (socket says 172.21.15.104:35904) 2024-02-12T16:28:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.830+0000 7fbae659c700 1 -- 172.21.15.104:0/2383602693 learned_addr learned my addr 172.21.15.104:0/2383602693 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.831+0000 7fbae659c700 1 -- 172.21.15.104:0/2383602693 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae80ff0c0 con 0x7fbae80703f0 2024-02-12T16:28:46.410 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.831+0000 7fbae659c700 1 --2- 172.21.15.104:0/2383602693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae80fea90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fbadc00b5f0 tx=0x7fbadc006ee0).ready entity=mon.0 client_cookie=caed04e4f207f9d6 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.831+0000 7fbae559a700 1 -- 172.21.15.104:0/2383602693 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbadc00d600 con 0x7fbae80703f0 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.831+0000 7fbae559a700 1 -- 172.21.15.104:0/2383602693 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbadc00dbf0 con 0x7fbae80703f0 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.831+0000 7fbae559a700 1 -- 172.21.15.104:0/2383602693 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbadc011ce0 con 0x7fbae80703f0 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.831+0000 7fbaece10700 1 -- 172.21.15.104:0/2383602693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 msgr2=0x7fbae80fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.831+0000 7fbaece10700 1 --2- 172.21.15.104:0/2383602693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae80fea90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fbadc00b5f0 tx=0x7fbadc006ee0).stop 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbaece10700 1 -- 172.21.15.104:0/2383602693 shutdown_connections 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbaece10700 1 --2- 172.21.15.104:0/2383602693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae80fea90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbaece10700 1 -- 172.21.15.104:0/2383602693 >> 172.21.15.104:0/2383602693 conn(0x7fbae80f6530 msgr2=0x7fbae80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbaece10700 1 -- 172.21.15.104:0/2383602693 shutdown_connections 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbaece10700 1 -- 172.21.15.104:0/2383602693 wait complete. 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbaece10700 1 Processor -- start 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbaece10700 1 -- start start 2024-02-12T16:28:46.411 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbaece10700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae810cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbaece10700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae810d290 con 0x7fbae80703f0 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbae659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae810cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbae659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae810cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35916/0 (socket says 172.21.15.104:35916) 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.832+0000 7fbae659c700 1 -- 172.21.15.104:0/1063292260 learned_addr learned my addr 172.21.15.104:0/1063292260 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.833+0000 7fbae659c700 1 -- 172.21.15.104:0/1063292260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc00b200 con 0x7fbae80703f0 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.833+0000 7fbae659c700 1 --2- 172.21.15.104:0/1063292260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae810cda0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fbadc011ed0 tx=0x7fbadc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.833+0000 7fbad6ffd700 1 -- 172.21.15.104:0/1063292260 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbadc01b570 con 0x7fbae80703f0 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.833+0000 7fbad6ffd700 1 -- 172.21.15.104:0/1063292260 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbadc01bb60 con 0x7fbae80703f0 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.833+0000 7fbaece10700 1 -- 172.21.15.104:0/1063292260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae810d490 con 0x7fbae80703f0 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.833+0000 7fbad6ffd700 1 -- 172.21.15.104:0/1063292260 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbadc01aad0 con 0x7fbae80703f0 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.833+0000 7fbaece10700 1 -- 172.21.15.104:0/1063292260 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae810e0d0 con 0x7fbae80703f0 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.834+0000 7fbad6ffd700 1 -- 172.21.15.104:0/1063292260 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fbadc00f070 con 0x7fbae80703f0 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.834+0000 7fbad6ffd700 1 --2- 172.21.15.104:0/1063292260 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fbacc02fd80 0x7fbacc032240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:46.412 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.834+0000 7fbad6ffd700 1 -- 172.21.15.104:0/1063292260 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fbadc0412b0 con 0x7fbae80703f0 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.834+0000 7fbaece10700 1 -- 172.21.15.104:0/1063292260 --> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbae8059100 con 0x7fbacc02fd80 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.834+0000 7fbae5d9b700 1 --2- 172.21.15.104:0/1063292260 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fbacc02fd80 0x7fbacc032240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.834+0000 7fbae5d9b700 1 --2- 172.21.15.104:0/1063292260 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fbacc02fd80 0x7fbacc032240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbad80096a0 tx=0x7fbad8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.835+0000 7fbad6ffd700 1 -- 172.21.15.104:0/1063292260 <== mgr.14114 v2:172.21.15.104:6800/3228071818 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fbae8059100 con 0x7fbacc02fd80 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.840+0000 7fbaece10700 1 -- 172.21.15.104:0/1063292260 --> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbae810d920 con 0x7fbacc02fd80 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.840+0000 7fbad6ffd700 1 -- 172.21.15.104:0/1063292260 <== mgr.14114 v2:172.21.15.104:6800/3228071818 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fbae810d920 con 0x7fbacc02fd80 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.840+0000 7fbaece10700 1 -- 172.21.15.104:0/1063292260 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fbacc02fd80 msgr2=0x7fbacc032240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.840+0000 7fbaece10700 1 --2- 172.21.15.104:0/1063292260 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fbacc02fd80 0x7fbacc032240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbad80096a0 tx=0x7fbad8006b40).stop 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.840+0000 7fbaece10700 1 -- 172.21.15.104:0/1063292260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 msgr2=0x7fbae810cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.840+0000 7fbaece10700 1 --2- 172.21.15.104:0/1063292260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae810cda0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fbadc011ed0 tx=0x7fbadc006ee0).stop 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.841+0000 7fbaece10700 1 -- 172.21.15.104:0/1063292260 shutdown_connections 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.841+0000 7fbaece10700 1 --2- 172.21.15.104:0/1063292260 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fbacc02fd80 0x7fbacc032240 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:46.413 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.841+0000 7fbaece10700 1 --2- 172.21.15.104:0/1063292260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbae80703f0 0x7fbae810cda0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.841+0000 7fbaece10700 1 -- 172.21.15.104:0/1063292260 >> 172.21.15.104:0/1063292260 conn(0x7fbae80f6530 msgr2=0x7fbae80fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.841+0000 7fbaece10700 1 -- 172.21.15.104:0/1063292260 shutdown_connections 2024-02-12T16:28:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:45.841+0000 7fbaece10700 1 -- 172.21.15.104:0/1063292260 wait complete. 2024-02-12T16:28:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-02-12T16:28:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-12T16:28:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "initialized": true 2024-02-12T16:28:46.414 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-02-12T16:28:47.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:45.835908+0000 mgr.smithi104.ohwbhf (mgr.14114) 1 : audit [DBG] from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-12T16:28:47.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:45.841724+0000 mgr.smithi104.ohwbhf (mgr.14114) 2 : audit [DBG] from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-12T16:28:47.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:28:45.909211+0000 mon.smithi104 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi104.ohwbhf(active, since 2s) 2024-02-12T16:28:49.915 INFO:teuthology.orchestra.run.smithi104.stderr:mgr epoch 5 is available 2024-02-12T16:28:49.915 INFO:teuthology.orchestra.run.smithi104.stderr:Setting orchestrator backend to cephadm... 2024-02-12T16:28:49.915 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-12T16:28:50.522 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.518+0000 7fe2bbac3700 1 Processor -- start 2024-02-12T16:28:50.522 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.519+0000 7fe2bbac3700 1 -- start start 2024-02-12T16:28:50.522 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.519+0000 7fe2bbac3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b40fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:50.522 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.519+0000 7fe2bbac3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b40fef80 con 0x7fe2b40703f0 2024-02-12T16:28:50.522 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.519+0000 7fe2b985f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b40fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:50.523 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.519+0000 7fe2b985f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b40fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35924/0 (socket says 172.21.15.104:35924) 2024-02-12T16:28:50.523 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.519+0000 7fe2b985f700 1 -- 172.21.15.104:0/2774485449 learned_addr learned my addr 172.21.15.104:0/2774485449 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:50.523 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.519+0000 7fe2b985f700 1 -- 172.21.15.104:0/2774485449 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40ff0c0 con 0x7fe2b40703f0 2024-02-12T16:28:50.523 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.519+0000 7fe2b985f700 1 --2- 172.21.15.104:0/2774485449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b40fea90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fe2a400b5f0 tx=0x7fe2a4006ee0).ready entity=mon.0 client_cookie=7ef2dfcbfabbdf6e server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:50.523 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.520+0000 7fe2b885d700 1 -- 172.21.15.104:0/2774485449 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a400d600 con 0x7fe2b40703f0 2024-02-12T16:28:50.523 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.520+0000 7fe2b885d700 1 -- 172.21.15.104:0/2774485449 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe2a400dbf0 con 0x7fe2b40703f0 2024-02-12T16:28:50.523 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.520+0000 7fe2b885d700 1 -- 172.21.15.104:0/2774485449 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a4011ce0 con 0x7fe2b40703f0 2024-02-12T16:28:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.520+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2774485449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 msgr2=0x7fe2b40fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.520+0000 7fe2bbac3700 1 --2- 172.21.15.104:0/2774485449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b40fea90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fe2a400b5f0 tx=0x7fe2a4006ee0).stop 2024-02-12T16:28:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.520+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2774485449 shutdown_connections 2024-02-12T16:28:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.520+0000 7fe2bbac3700 1 --2- 172.21.15.104:0/2774485449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b40fea90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.520+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2774485449 >> 172.21.15.104:0/2774485449 conn(0x7fe2b40f6530 msgr2=0x7fe2b40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.520+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2774485449 shutdown_connections 2024-02-12T16:28:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.521+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2774485449 wait complete. 2024-02-12T16:28:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.521+0000 7fe2bbac3700 1 Processor -- start 2024-02-12T16:28:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.521+0000 7fe2bbac3700 1 -- start start 2024-02-12T16:28:50.525 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.521+0000 7fe2bbac3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.521+0000 7fe2bbac3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b410b080 con 0x7fe2b40703f0 2024-02-12T16:28:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.522+0000 7fe2b985f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.522+0000 7fe2b985f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b410ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35940/0 (socket says 172.21.15.104:35940) 2024-02-12T16:28:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.522+0000 7fe2b985f700 1 -- 172.21.15.104:0/2230380650 learned_addr learned my addr 172.21.15.104:0/2230380650 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.522+0000 7fe2b985f700 1 -- 172.21.15.104:0/2230380650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a400b200 con 0x7fe2b40703f0 2024-02-12T16:28:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.522+0000 7fe2b985f700 1 --2- 172.21.15.104:0/2230380650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b410ab90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fe2a40115c0 tx=0x7fe2a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.522+0000 7fe2a27fc700 1 -- 172.21.15.104:0/2230380650 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a401b410 con 0x7fe2b40703f0 2024-02-12T16:28:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.522+0000 7fe2a27fc700 1 -- 172.21.15.104:0/2230380650 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe2a401ba00 con 0x7fe2b40703f0 2024-02-12T16:28:50.526 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.522+0000 7fe2a27fc700 1 -- 172.21.15.104:0/2230380650 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a401aaa0 con 0x7fe2b40703f0 2024-02-12T16:28:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.522+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2230380650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b410b280 con 0x7fe2b40703f0 2024-02-12T16:28:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.522+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2230380650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b410b710 con 0x7fe2b40703f0 2024-02-12T16:28:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.523+0000 7fe2a27fc700 1 -- 172.21.15.104:0/2230380650 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe2a400f070 con 0x7fe2b40703f0 2024-02-12T16:28:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.523+0000 7fe2a27fc700 1 --2- 172.21.15.104:0/2230380650 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fe29c02fe70 0x7fe29c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.523+0000 7fe2a27fc700 1 -- 172.21.15.104:0/2230380650 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe2a4040ff0 con 0x7fe2b40703f0 2024-02-12T16:28:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.523+0000 7fe2b905e700 1 --2- 172.21.15.104:0/2230380650 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fe29c02fe70 0x7fe29c032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.523+0000 7fe2b905e700 1 --2- 172.21.15.104:0/2230380650 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fe29c02fe70 0x7fe29c032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe2b00096a0 tx=0x7fe2b0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:50.527 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.524+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2230380650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b4045ad0 con 0x7fe2b40703f0 2024-02-12T16:28:50.530 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.528+0000 7fe2a27fc700 1 -- 172.21.15.104:0/2230380650 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2a4016070 con 0x7fe2b40703f0 2024-02-12T16:28:50.708 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.705+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2230380650 --> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2b4104c00 con 0x7fe29c02fe70 2024-02-12T16:28:50.712 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.710+0000 7fe2a27fc700 1 -- 172.21.15.104:0/2230380650 <== mgr.14114 v2:172.21.15.104:6800/3228071818 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe2b4104c00 con 0x7fe29c02fe70 2024-02-12T16:28:50.715 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.712+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2230380650 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fe29c02fe70 msgr2=0x7fe29c032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:50.716 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.712+0000 7fe2bbac3700 1 --2- 172.21.15.104:0/2230380650 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fe29c02fe70 0x7fe29c032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe2b00096a0 tx=0x7fe2b0006b40).stop 2024-02-12T16:28:50.716 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.712+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2230380650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 msgr2=0x7fe2b410ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:50.716 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.712+0000 7fe2bbac3700 1 --2- 172.21.15.104:0/2230380650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b410ab90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fe2a40115c0 tx=0x7fe2a4006ee0).stop 2024-02-12T16:28:50.716 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.712+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2230380650 shutdown_connections 2024-02-12T16:28:50.716 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.712+0000 7fe2bbac3700 1 --2- 172.21.15.104:0/2230380650 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fe29c02fe70 0x7fe29c032330 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:50.716 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.712+0000 7fe2bbac3700 1 --2- 172.21.15.104:0/2230380650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe2b40703f0 0x7fe2b410ab90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:50.716 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.712+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2230380650 >> 172.21.15.104:0/2230380650 conn(0x7fe2b40f6530 msgr2=0x7fe2b40f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:50.716 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.712+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2230380650 shutdown_connections 2024-02-12T16:28:50.716 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:50.712+0000 7fe2bbac3700 1 -- 172.21.15.104:0/2230380650 wait complete. 2024-02-12T16:28:51.279 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-12T16:28:51.831 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.816+0000 7f4d38944700 1 Processor -- start 2024-02-12T16:28:51.831 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.816+0000 7f4d38944700 1 -- start start 2024-02-12T16:28:51.831 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.817+0000 7f4d38944700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d34100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:51.831 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.817+0000 7f4d38944700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d340fc8e0 con 0x7f4d34070390 2024-02-12T16:28:51.831 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.817+0000 7f4d3259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d34100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:51.832 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.817+0000 7f4d3259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d34100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35950/0 (socket says 172.21.15.104:35950) 2024-02-12T16:28:51.832 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.817+0000 7f4d3259c700 1 -- 172.21.15.104:0/1950692020 learned_addr learned my addr 172.21.15.104:0/1950692020 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:51.832 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.817+0000 7f4d3259c700 1 -- 172.21.15.104:0/1950692020 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d341013e0 con 0x7f4d34070390 2024-02-12T16:28:51.832 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.817+0000 7f4d3259c700 1 --2- 172.21.15.104:0/1950692020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d34100ef0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f4d2800b5f0 tx=0x7f4d28006ee0).ready entity=mon.0 client_cookie=1eb7835cc944f10b server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:51.832 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.817+0000 7f4d3159a700 1 -- 172.21.15.104:0/1950692020 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d2800d600 con 0x7f4d34070390 2024-02-12T16:28:51.832 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.817+0000 7f4d3159a700 1 -- 172.21.15.104:0/1950692020 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4d2800dbf0 con 0x7f4d34070390 2024-02-12T16:28:51.832 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.817+0000 7f4d3159a700 1 -- 172.21.15.104:0/1950692020 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d28011cb0 con 0x7f4d34070390 2024-02-12T16:28:51.832 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.818+0000 7f4d38944700 1 -- 172.21.15.104:0/1950692020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 msgr2=0x7f4d34100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:51.832 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.818+0000 7f4d38944700 1 --2- 172.21.15.104:0/1950692020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d34100ef0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f4d2800b5f0 tx=0x7f4d28006ee0).stop 2024-02-12T16:28:51.832 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.818+0000 7f4d38944700 1 -- 172.21.15.104:0/1950692020 shutdown_connections 2024-02-12T16:28:51.833 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.818+0000 7f4d38944700 1 --2- 172.21.15.104:0/1950692020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d34100ef0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:51.833 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.818+0000 7f4d38944700 1 -- 172.21.15.104:0/1950692020 >> 172.21.15.104:0/1950692020 conn(0x7f4d340f6530 msgr2=0x7f4d340f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:51.833 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.818+0000 7f4d38944700 1 -- 172.21.15.104:0/1950692020 shutdown_connections 2024-02-12T16:28:51.833 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.818+0000 7f4d38944700 1 -- 172.21.15.104:0/1950692020 wait complete. 2024-02-12T16:28:51.833 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.818+0000 7f4d38944700 1 Processor -- start 2024-02-12T16:28:51.833 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.818+0000 7f4d38944700 1 -- start start 2024-02-12T16:28:51.833 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d38944700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d3410cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:51.833 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d38944700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3410d270 con 0x7f4d34070390 2024-02-12T16:28:51.833 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d3259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d3410cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:51.833 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d3259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d3410cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:35954/0 (socket says 172.21.15.104:35954) 2024-02-12T16:28:51.833 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d3259c700 1 -- 172.21.15.104:0/3692226573 learned_addr learned my addr 172.21.15.104:0/3692226573 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d3259c700 1 -- 172.21.15.104:0/3692226573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2800b200 con 0x7f4d34070390 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d3259c700 1 --2- 172.21.15.104:0/3692226573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d3410cd80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4d28006800 tx=0x7f4d28006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d22ffd700 1 -- 172.21.15.104:0/3692226573 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d2801b570 con 0x7f4d34070390 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d22ffd700 1 -- 172.21.15.104:0/3692226573 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4d2801bb60 con 0x7f4d34070390 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d22ffd700 1 -- 172.21.15.104:0/3692226573 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d2801a9f0 con 0x7f4d34070390 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d38944700 1 -- 172.21.15.104:0/3692226573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d3410d470 con 0x7f4d34070390 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.819+0000 7f4d38944700 1 -- 172.21.15.104:0/3692226573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d3410e080 con 0x7f4d34070390 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.820+0000 7f4d22ffd700 1 -- 172.21.15.104:0/3692226573 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4d2800f070 con 0x7f4d34070390 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.820+0000 7f4d22ffd700 1 --2- 172.21.15.104:0/3692226573 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f4d1802fe70 0x7f4d18032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.820+0000 7f4d22ffd700 1 -- 172.21.15.104:0/3692226573 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4d28041350 con 0x7f4d34070390 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.820+0000 7f4d38944700 1 -- 172.21.15.104:0/3692226573 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d34059100 con 0x7f4d34070390 2024-02-12T16:28:51.834 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.820+0000 7f4d31d9b700 1 --2- 172.21.15.104:0/3692226573 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f4d1802fe70 0x7f4d18032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:51.835 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.820+0000 7f4d31d9b700 1 --2- 172.21.15.104:0/3692226573 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f4d1802fe70 0x7f4d18032330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4d240096a0 tx=0x7f4d24006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:51.835 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.824+0000 7f4d22ffd700 1 -- 172.21.15.104:0/3692226573 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d28016070 con 0x7f4d34070390 2024-02-12T16:28:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.976+0000 7f4d38944700 1 -- 172.21.15.104:0/3692226573 --> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d34102140 con 0x7f4d1802fe70 2024-02-12T16:28:51.978 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.976+0000 7f4d22ffd700 1 -- 172.21.15.104:0/3692226573 <== mgr.14114 v2:172.21.15.104:6800/3228071818 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f4d34102140 con 0x7f4d1802fe70 2024-02-12T16:28:51.979 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: value unchanged 2024-02-12T16:28:51.981 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.978+0000 7f4d38944700 1 -- 172.21.15.104:0/3692226573 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f4d1802fe70 msgr2=0x7f4d18032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:51.981 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.978+0000 7f4d38944700 1 --2- 172.21.15.104:0/3692226573 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f4d1802fe70 0x7f4d18032330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4d240096a0 tx=0x7f4d24006b40).stop 2024-02-12T16:28:51.982 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.978+0000 7f4d38944700 1 -- 172.21.15.104:0/3692226573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 msgr2=0x7f4d3410cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:51.982 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.978+0000 7f4d38944700 1 --2- 172.21.15.104:0/3692226573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d3410cd80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4d28006800 tx=0x7f4d28006ee0).stop 2024-02-12T16:28:51.982 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.978+0000 7f4d38944700 1 -- 172.21.15.104:0/3692226573 shutdown_connections 2024-02-12T16:28:51.982 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.978+0000 7f4d38944700 1 --2- 172.21.15.104:0/3692226573 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f4d1802fe70 0x7f4d18032330 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:51.982 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.978+0000 7f4d38944700 1 --2- 172.21.15.104:0/3692226573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d34070390 0x7f4d3410cd80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:51.982 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.978+0000 7f4d38944700 1 -- 172.21.15.104:0/3692226573 >> 172.21.15.104:0/3692226573 conn(0x7f4d340f6530 msgr2=0x7f4d340f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:51.982 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.978+0000 7f4d38944700 1 -- 172.21.15.104:0/3692226573 shutdown_connections 2024-02-12T16:28:51.982 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:51.978+0000 7f4d38944700 1 -- 172.21.15.104:0/3692226573 wait complete. 2024-02-12T16:28:51.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:51 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:50.707687+0000 mgr.smithi104.ohwbhf (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:28:51.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:51 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:50.711487+0000 mon.smithi104 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:28:51.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:51 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:50.714910+0000 mon.smithi104 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:28:52.643 INFO:teuthology.orchestra.run.smithi104.stderr:Generating ssh key... 2024-02-12T16:28:52.644 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-12T16:28:53.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:53 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:51.977718+0000 mgr.smithi104.ohwbhf (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:28:53.218 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.214+0000 7f9141587700 1 Processor -- start 2024-02-12T16:28:53.218 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.214+0000 7f9141587700 1 -- start start 2024-02-12T16:28:53.218 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.215+0000 7f9141587700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c0fc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:53.218 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.215+0000 7f9141587700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0fc790 con 0x7f913c0fbe60 2024-02-12T16:28:53.218 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.215+0000 7f913affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c0fc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:53.218 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.215+0000 7f913affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c0fc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41386/0 (socket says 172.21.15.104:41386) 2024-02-12T16:28:53.218 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.215+0000 7f913affd700 1 -- 172.21.15.104:0/194493743 learned_addr learned my addr 172.21.15.104:0/194493743 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:53.218 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.215+0000 7f913affd700 1 -- 172.21.15.104:0/194493743 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0fc8d0 con 0x7f913c0fbe60 2024-02-12T16:28:53.221 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.215+0000 7f913affd700 1 --2- 172.21.15.104:0/194493743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c0fc2a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f912400b5f0 tx=0x7f9124006ee0).ready entity=mon.0 client_cookie=31bceaff0cdd2107 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:53.221 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.215+0000 7f9139ffb700 1 -- 172.21.15.104:0/194493743 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f912400d600 con 0x7f913c0fbe60 2024-02-12T16:28:53.221 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.215+0000 7f9139ffb700 1 -- 172.21.15.104:0/194493743 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f912400dbf0 con 0x7f913c0fbe60 2024-02-12T16:28:53.221 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.215+0000 7f9139ffb700 1 -- 172.21.15.104:0/194493743 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9124011cb0 con 0x7f913c0fbe60 2024-02-12T16:28:53.221 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.216+0000 7f9141587700 1 -- 172.21.15.104:0/194493743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 msgr2=0x7f913c0fc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:53.221 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.216+0000 7f9141587700 1 --2- 172.21.15.104:0/194493743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c0fc2a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f912400b5f0 tx=0x7f9124006ee0).stop 2024-02-12T16:28:53.221 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.216+0000 7f9141587700 1 -- 172.21.15.104:0/194493743 shutdown_connections 2024-02-12T16:28:53.221 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.216+0000 7f9141587700 1 --2- 172.21.15.104:0/194493743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c0fc2a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:53.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.216+0000 7f9141587700 1 -- 172.21.15.104:0/194493743 >> 172.21.15.104:0/194493743 conn(0x7f913c0f7670 msgr2=0x7f913c0f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:53.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.216+0000 7f9141587700 1 -- 172.21.15.104:0/194493743 shutdown_connections 2024-02-12T16:28:53.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.216+0000 7f9141587700 1 -- 172.21.15.104:0/194493743 wait complete. 2024-02-12T16:28:53.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.216+0000 7f9141587700 1 Processor -- start 2024-02-12T16:28:53.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f9141587700 1 -- start start 2024-02-12T16:28:53.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f9141587700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c10ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:53.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f9141587700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c10b040 con 0x7f913c0fbe60 2024-02-12T16:28:53.223 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f913affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c10ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:53.223 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f913affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c10ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41402/0 (socket says 172.21.15.104:41402) 2024-02-12T16:28:53.223 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f913affd700 1 -- 172.21.15.104:0/3411981438 learned_addr learned my addr 172.21.15.104:0/3411981438 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:53.223 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f913affd700 1 -- 172.21.15.104:0/3411981438 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912400b200 con 0x7f913c0fbe60 2024-02-12T16:28:53.223 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f913affd700 1 --2- 172.21.15.104:0/3411981438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c10ab50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f9124006800 tx=0x7f9124006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:53.223 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f91337fe700 1 -- 172.21.15.104:0/3411981438 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f912401b570 con 0x7f913c0fbe60 2024-02-12T16:28:53.223 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f91337fe700 1 -- 172.21.15.104:0/3411981438 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f912401bb60 con 0x7f913c0fbe60 2024-02-12T16:28:53.223 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f9141587700 1 -- 172.21.15.104:0/3411981438 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c10b240 con 0x7f913c0fbe60 2024-02-12T16:28:53.223 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f91337fe700 1 -- 172.21.15.104:0/3411981438 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f912401a9f0 con 0x7f913c0fbe60 2024-02-12T16:28:53.223 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.217+0000 7f9141587700 1 -- 172.21.15.104:0/3411981438 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c10b6d0 con 0x7f913c0fbe60 2024-02-12T16:28:53.224 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.218+0000 7f91337fe700 1 -- 172.21.15.104:0/3411981438 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f912400f070 con 0x7f913c0fbe60 2024-02-12T16:28:53.224 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.218+0000 7f91337fe700 1 --2- 172.21.15.104:0/3411981438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f911c02fe70 0x7f911c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:53.224 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.218+0000 7f91337fe700 1 -- 172.21.15.104:0/3411981438 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9124042210 con 0x7f913c0fbe60 2024-02-12T16:28:53.224 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.218+0000 7f9141587700 1 -- 172.21.15.104:0/3411981438 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c045ad0 con 0x7f913c0fbe60 2024-02-12T16:28:53.224 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.218+0000 7f913a7fc700 1 --2- 172.21.15.104:0/3411981438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f911c02fe70 0x7f911c032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:53.224 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.219+0000 7f913a7fc700 1 --2- 172.21.15.104:0/3411981438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f911c02fe70 0x7f911c032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f912c0096a0 tx=0x7f912c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:53.225 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.223+0000 7f91337fe700 1 -- 172.21.15.104:0/3411981438 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9124016070 con 0x7f913c0fbe60 2024-02-12T16:28:53.374 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.372+0000 7f9141587700 1 -- 172.21.15.104:0/3411981438 --> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f913c104bf0 con 0x7f911c02fe70 2024-02-12T16:28:53.553 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.550+0000 7f91337fe700 1 -- 172.21.15.104:0/3411981438 <== mgr.14114 v2:172.21.15.104:6800/3228071818 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f913c104bf0 con 0x7f911c02fe70 2024-02-12T16:28:53.555 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.552+0000 7f9141587700 1 -- 172.21.15.104:0/3411981438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f911c02fe70 msgr2=0x7f911c032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:53.556 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.552+0000 7f9141587700 1 --2- 172.21.15.104:0/3411981438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f911c02fe70 0x7f911c032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f912c0096a0 tx=0x7f912c006b40).stop 2024-02-12T16:28:53.556 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.552+0000 7f9141587700 1 -- 172.21.15.104:0/3411981438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 msgr2=0x7f913c10ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:53.556 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.552+0000 7f9141587700 1 --2- 172.21.15.104:0/3411981438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c10ab50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f9124006800 tx=0x7f9124006ee0).stop 2024-02-12T16:28:53.556 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.552+0000 7f9141587700 1 -- 172.21.15.104:0/3411981438 shutdown_connections 2024-02-12T16:28:53.556 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.552+0000 7f9141587700 1 --2- 172.21.15.104:0/3411981438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f911c02fe70 0x7f911c032330 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:53.556 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.552+0000 7f9141587700 1 --2- 172.21.15.104:0/3411981438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f913c0fbe60 0x7f913c10ab50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:53.556 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.552+0000 7f9141587700 1 -- 172.21.15.104:0/3411981438 >> 172.21.15.104:0/3411981438 conn(0x7f913c0f7670 msgr2=0x7f913c0f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:53.556 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.552+0000 7f9141587700 1 -- 172.21.15.104:0/3411981438 shutdown_connections 2024-02-12T16:28:53.556 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:53.552+0000 7f9141587700 1 -- 172.21.15.104:0/3411981438 wait complete. 2024-02-12T16:28:54.208 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-12T16:28:54.877 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.872+0000 7f7d3bf81700 1 Processor -- start 2024-02-12T16:28:54.878 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.872+0000 7f7d3bf81700 1 -- start start 2024-02-12T16:28:54.878 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.872+0000 7f7d3bf81700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d340fd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:54.878 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.872+0000 7f7d3bf81700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d340fde70 con 0x7f7d340fd560 2024-02-12T16:28:54.878 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.872+0000 7f7d39d1d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d340fd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:54.878 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.873+0000 7f7d39d1d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d340fd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41404/0 (socket says 172.21.15.104:41404) 2024-02-12T16:28:54.878 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.873+0000 7f7d39d1d700 1 -- 172.21.15.104:0/1641192954 learned_addr learned my addr 172.21.15.104:0/1641192954 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:54.878 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.873+0000 7f7d39d1d700 1 -- 172.21.15.104:0/1641192954 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d340fdfb0 con 0x7f7d340fd560 2024-02-12T16:28:54.878 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.873+0000 7f7d39d1d700 1 --2- 172.21.15.104:0/1641192954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d340fd980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f7d2400b5f0 tx=0x7f7d24006ee0).ready entity=mon.0 client_cookie=69d34aaab64a8994 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:54.878 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.873+0000 7f7d38d1b700 1 -- 172.21.15.104:0/1641192954 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d2400d600 con 0x7f7d340fd560 2024-02-12T16:28:54.878 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.873+0000 7f7d38d1b700 1 -- 172.21.15.104:0/1641192954 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7d2400dbf0 con 0x7f7d340fd560 2024-02-12T16:28:54.879 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.873+0000 7f7d38d1b700 1 -- 172.21.15.104:0/1641192954 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d24011cb0 con 0x7f7d340fd560 2024-02-12T16:28:54.879 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.873+0000 7f7d3bf81700 1 -- 172.21.15.104:0/1641192954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 msgr2=0x7f7d340fd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:54.879 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.873+0000 7f7d3bf81700 1 --2- 172.21.15.104:0/1641192954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d340fd980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f7d2400b5f0 tx=0x7f7d24006ee0).stop 2024-02-12T16:28:54.879 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.873+0000 7f7d3bf81700 1 -- 172.21.15.104:0/1641192954 shutdown_connections 2024-02-12T16:28:54.879 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.873+0000 7f7d3bf81700 1 --2- 172.21.15.104:0/1641192954 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d340fd980 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:54.879 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.874+0000 7f7d3bf81700 1 -- 172.21.15.104:0/1641192954 >> 172.21.15.104:0/1641192954 conn(0x7f7d340f5d20 msgr2=0x7f7d340f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.874+0000 7f7d3bf81700 1 -- 172.21.15.104:0/1641192954 shutdown_connections 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.874+0000 7f7d3bf81700 1 -- 172.21.15.104:0/1641192954 wait complete. 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.874+0000 7f7d3bf81700 1 Processor -- start 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.874+0000 7f7d3bf81700 1 -- start start 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.874+0000 7f7d3bf81700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d3410abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.874+0000 7f7d3bf81700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3410b0d0 con 0x7f7d340fd560 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.874+0000 7f7d39d1d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d3410abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.874+0000 7f7d39d1d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d3410abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41420/0 (socket says 172.21.15.104:41420) 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.874+0000 7f7d39d1d700 1 -- 172.21.15.104:0/3129546216 learned_addr learned my addr 172.21.15.104:0/3129546216 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.875+0000 7f7d39d1d700 1 -- 172.21.15.104:0/3129546216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2400b200 con 0x7f7d340fd560 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.875+0000 7f7d39d1d700 1 --2- 172.21.15.104:0/3129546216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d3410abe0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f7d24002f40 tx=0x7f7d24006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.875+0000 7f7d227fc700 1 -- 172.21.15.104:0/3129546216 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d2401b410 con 0x7f7d340fd560 2024-02-12T16:28:54.880 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.875+0000 7f7d227fc700 1 -- 172.21.15.104:0/3129546216 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7d2401ba00 con 0x7f7d340fd560 2024-02-12T16:28:54.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.875+0000 7f7d227fc700 1 -- 172.21.15.104:0/3129546216 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d2401a900 con 0x7f7d340fd560 2024-02-12T16:28:54.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.875+0000 7f7d3bf81700 1 -- 172.21.15.104:0/3129546216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3410b2d0 con 0x7f7d340fd560 2024-02-12T16:28:54.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.875+0000 7f7d3bf81700 1 -- 172.21.15.104:0/3129546216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3410b760 con 0x7f7d340fd560 2024-02-12T16:28:54.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.876+0000 7f7d227fc700 1 -- 172.21.15.104:0/3129546216 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f7d2400f070 con 0x7f7d340fd560 2024-02-12T16:28:54.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.876+0000 7f7d3bf81700 1 -- 172.21.15.104:0/3129546216 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d34045ad0 con 0x7f7d340fd560 2024-02-12T16:28:54.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.876+0000 7f7d227fc700 1 --2- 172.21.15.104:0/3129546216 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f7d1c02fe10 0x7f7d1c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:54.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.876+0000 7f7d227fc700 1 -- 172.21.15.104:0/3129546216 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7d240413d0 con 0x7f7d340fd560 2024-02-12T16:28:54.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.876+0000 7f7d3951c700 1 --2- 172.21.15.104:0/3129546216 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f7d1c02fe10 0x7f7d1c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:54.881 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.877+0000 7f7d3951c700 1 --2- 172.21.15.104:0/3129546216 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f7d1c02fe10 0x7f7d1c0322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7d300096a0 tx=0x7f7d30006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:54.882 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:54.880+0000 7f7d227fc700 1 -- 172.21.15.104:0/3129546216 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d24022310 con 0x7f7d340fd560 2024-02-12T16:28:54.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:28:53.374303+0000 mgr.smithi104.ohwbhf (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-12T16:28:54.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:53.548821+0000 mon.smithi104 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:28:54.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:53.551172+0000 mon.smithi104 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:28:55.036 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.033+0000 7f7d3bf81700 1 -- 172.21.15.104:0/3129546216 --> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d3410bfc0 con 0x7f7d1c02fe10 2024-02-12T16:28:55.036 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.034+0000 7f7d227fc700 1 -- 172.21.15.104:0/3129546216 <== mgr.14114 v2:172.21.15.104:6800/3228071818 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f7d3410bfc0 con 0x7f7d1c02fe10 2024-02-12T16:28:55.036 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJf7a0S5c9C7Mq6QgVacJq3Uis8FgkK7Z3ogsBkGGxLJOMKz4zVB+j3F6zYRChcr0BQ+3PZVZuSofNO3AYC6DPEfTsHwLnUCkQtSds5By+VaM4KeAJCykmxVrH5K88staZMpbnXtWPdjgFfsNrDuZtYHPz3SqTkdKjqgFz6uaiJqhejjmIwlbBvuADuAktoH+zMdQWRCyup+cIlEm8GL8fvXlYyM0gKhTbfvpC9trY3H8TddI7PoVlJgyaqMOO9HrabO3Nw6cHTMLbAHJNJInQUxXj4xPiBtm8blUnjE+qr5RFn5TXt3dukgVtUVu3dWnEtHFQ7VgmwbGpnuDvoYLIBr3WtKOOM+0DLKSlzXhmhKayZkZJoDk7nE0Tk2SWJXwGdLCrDYKD+m0dbkmrraGjuAX+lnyci7shfYS6AYkxQpjmTaWp7t72Q8OBPlqsSpsyEX0MjgshbDofh4xW/XiCdfMiocZcoGukPzyjWNUezlMjuJ3suDNJx0sRH4+iwIc= ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:28:55.038 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.035+0000 7f7d3bf81700 1 -- 172.21.15.104:0/3129546216 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f7d1c02fe10 msgr2=0x7f7d1c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:55.039 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.035+0000 7f7d3bf81700 1 --2- 172.21.15.104:0/3129546216 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f7d1c02fe10 0x7f7d1c0322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7d300096a0 tx=0x7f7d30006b40).stop 2024-02-12T16:28:55.039 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.035+0000 7f7d3bf81700 1 -- 172.21.15.104:0/3129546216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 msgr2=0x7f7d3410abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:55.039 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.035+0000 7f7d3bf81700 1 --2- 172.21.15.104:0/3129546216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d3410abe0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f7d24002f40 tx=0x7f7d24006ee0).stop 2024-02-12T16:28:55.039 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.035+0000 7f7d3bf81700 1 -- 172.21.15.104:0/3129546216 shutdown_connections 2024-02-12T16:28:55.039 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.035+0000 7f7d3bf81700 1 --2- 172.21.15.104:0/3129546216 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f7d1c02fe10 0x7f7d1c0322d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:55.039 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.035+0000 7f7d3bf81700 1 --2- 172.21.15.104:0/3129546216 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7d340fd560 0x7f7d3410abe0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:55.039 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.035+0000 7f7d3bf81700 1 -- 172.21.15.104:0/3129546216 >> 172.21.15.104:0/3129546216 conn(0x7f7d340f5d20 msgr2=0x7f7d340ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:55.039 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.036+0000 7f7d3bf81700 1 -- 172.21.15.104:0/3129546216 shutdown_connections 2024-02-12T16:28:55.039 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:55.036+0000 7f7d3bf81700 1 -- 172.21.15.104:0/3129546216 wait complete. 2024-02-12T16:28:55.715 INFO:teuthology.orchestra.run.smithi104.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-12T16:28:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:Adding key to root@localhost authorized_keys... 2024-02-12T16:28:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:Adding host smithi104... 2024-02-12T16:28:55.716 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi104 172.21.15.104 2024-02-12T16:28:56.245 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.239+0000 7f82fbe5e700 1 Processor -- start 2024-02-12T16:28:56.245 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.239+0000 7f82fbe5e700 1 -- start start 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.239+0000 7f82fbe5e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40feb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.239+0000 7f82fbe5e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40ff040 con 0x7f82f406fbd0 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.240+0000 7f82f9bfa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40feb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.240+0000 7f82f9bfa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40feb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41422/0 (socket says 172.21.15.104:41422) 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.240+0000 7f82f9bfa700 1 -- 172.21.15.104:0/115306412 learned_addr learned my addr 172.21.15.104:0/115306412 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.240+0000 7f82f9bfa700 1 -- 172.21.15.104:0/115306412 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f40ff180 con 0x7f82f406fbd0 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.240+0000 7f82f9bfa700 1 --2- 172.21.15.104:0/115306412 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40feb50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f82e400b5f0 tx=0x7f82e4006ee0).ready entity=mon.0 client_cookie=b31c07761dee4f05 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.240+0000 7f82f8bf8700 1 -- 172.21.15.104:0/115306412 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e400d600 con 0x7f82f406fbd0 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.240+0000 7f82f8bf8700 1 -- 172.21.15.104:0/115306412 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f82e400dbf0 con 0x7f82f406fbd0 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.240+0000 7f82f8bf8700 1 -- 172.21.15.104:0/115306412 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e4011ce0 con 0x7f82f406fbd0 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.241+0000 7f82fbe5e700 1 -- 172.21.15.104:0/115306412 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 msgr2=0x7f82f40feb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:56.246 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.241+0000 7f82fbe5e700 1 --2- 172.21.15.104:0/115306412 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40feb50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f82e400b5f0 tx=0x7f82e4006ee0).stop 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.241+0000 7f82fbe5e700 1 -- 172.21.15.104:0/115306412 shutdown_connections 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.241+0000 7f82fbe5e700 1 --2- 172.21.15.104:0/115306412 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40feb50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.241+0000 7f82fbe5e700 1 -- 172.21.15.104:0/115306412 >> 172.21.15.104:0/115306412 conn(0x7f82f40f65b0 msgr2=0x7f82f40f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.241+0000 7f82fbe5e700 1 -- 172.21.15.104:0/115306412 shutdown_connections 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.241+0000 7f82fbe5e700 1 -- 172.21.15.104:0/115306412 wait complete. 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.241+0000 7f82fbe5e700 1 Processor -- start 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82fbe5e700 1 -- start start 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82fbe5e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40fc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82fbe5e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40faa50 con 0x7f82f406fbd0 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82f9bfa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40fc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82f9bfa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40fc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41428/0 (socket says 172.21.15.104:41428) 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82f9bfa700 1 -- 172.21.15.104:0/1729841438 learned_addr learned my addr 172.21.15.104:0/1729841438 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:56.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82f9bfa700 1 -- 172.21.15.104:0/1729841438 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e400b200 con 0x7f82f406fbd0 2024-02-12T16:28:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82f9bfa700 1 --2- 172.21.15.104:0/1729841438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40fc5d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f82e4006800 tx=0x7f82e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82e27fc700 1 -- 172.21.15.104:0/1729841438 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e401b570 con 0x7f82f406fbd0 2024-02-12T16:28:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82e27fc700 1 -- 172.21.15.104:0/1729841438 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f82e401bb60 con 0x7f82f406fbd0 2024-02-12T16:28:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82e27fc700 1 -- 172.21.15.104:0/1729841438 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e401a9f0 con 0x7f82f406fbd0 2024-02-12T16:28:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82fbe5e700 1 -- 172.21.15.104:0/1729841438 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f40fabf0 con 0x7f82f406fbd0 2024-02-12T16:28:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.242+0000 7f82fbe5e700 1 -- 172.21.15.104:0/1729841438 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f40fb080 con 0x7f82f406fbd0 2024-02-12T16:28:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.243+0000 7f82e27fc700 1 -- 172.21.15.104:0/1729841438 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f82e400f070 con 0x7f82f406fbd0 2024-02-12T16:28:56.248 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.243+0000 7f82e27fc700 1 --2- 172.21.15.104:0/1729841438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f82dc02fe70 0x7f82dc032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:56.249 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.243+0000 7f82e27fc700 1 -- 172.21.15.104:0/1729841438 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f82e4041400 con 0x7f82f406fbd0 2024-02-12T16:28:56.249 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.243+0000 7f82f93f9700 1 --2- 172.21.15.104:0/1729841438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f82dc02fe70 0x7f82dc032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:56.249 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.244+0000 7f82fbe5e700 1 -- 172.21.15.104:0/1729841438 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f4059100 con 0x7f82f406fbd0 2024-02-12T16:28:56.249 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.244+0000 7f82f93f9700 1 --2- 172.21.15.104:0/1729841438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f82dc02fe70 0x7f82dc032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f82f00096a0 tx=0x7f82f0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:56.250 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.248+0000 7f82e27fc700 1 -- 172.21.15.104:0/1729841438 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82e4016080 con 0x7f82f406fbd0 2024-02-12T16:28:56.421 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:56.418+0000 7f82fbe5e700 1 -- 172.21.15.104:0/1729841438 --> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi104", "addr": "172.21.15.104", "target": ["mon-mgr", ""]}) v1 -- 0x7f82f4104b50 con 0x7f82dc02fe70 2024-02-12T16:28:56.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:55.035134+0000 mgr.smithi104.ohwbhf (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:28:57.022 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Added host 'smithi104' 2024-02-12T16:28:57.023 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:57.019+0000 7f82e27fc700 1 -- 172.21.15.104:0/1729841438 <== mgr.14114 v2:172.21.15.104:6800/3228071818 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f82f4104b50 con 0x7f82dc02fe70 2024-02-12T16:28:57.024 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:57.022+0000 7f82fbe5e700 1 -- 172.21.15.104:0/1729841438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f82dc02fe70 msgr2=0x7f82dc032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:57.024 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:57.022+0000 7f82fbe5e700 1 --2- 172.21.15.104:0/1729841438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f82dc02fe70 0x7f82dc032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f82f00096a0 tx=0x7f82f0006b40).stop 2024-02-12T16:28:57.024 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:57.022+0000 7f82fbe5e700 1 -- 172.21.15.104:0/1729841438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 msgr2=0x7f82f40fc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:57.024 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:57.022+0000 7f82fbe5e700 1 --2- 172.21.15.104:0/1729841438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40fc5d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f82e4006800 tx=0x7f82e4006ee0).stop 2024-02-12T16:28:57.025 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:57.023+0000 7f82fbe5e700 1 -- 172.21.15.104:0/1729841438 shutdown_connections 2024-02-12T16:28:57.025 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:57.023+0000 7f82fbe5e700 1 --2- 172.21.15.104:0/1729841438 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f82dc02fe70 0x7f82dc032330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:57.025 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:57.023+0000 7f82fbe5e700 1 --2- 172.21.15.104:0/1729841438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82f406fbd0 0x7f82f40fc5d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:57.025 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:57.023+0000 7f82fbe5e700 1 -- 172.21.15.104:0/1729841438 >> 172.21.15.104:0/1729841438 conn(0x7f82f40f65b0 msgr2=0x7f82f40f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:57.026 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:57.023+0000 7f82fbe5e700 1 -- 172.21.15.104:0/1729841438 shutdown_connections 2024-02-12T16:28:57.026 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:57.023+0000 7f82fbe5e700 1 -- 172.21.15.104:0/1729841438 wait complete. 2024-02-12T16:28:57.344 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:57 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:56.420483+0000 mgr.smithi104.ohwbhf (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "addr": "172.21.15.104", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:28:57.344 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:57 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:57.019716+0000 mon.smithi104 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:28:57.345 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:57 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:57.020260+0000 mon.smithi104 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:28:57.605 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying mon service with default placement... 2024-02-12T16:28:57.605 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-12T16:28:58.190 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.141+0000 7fb769331700 1 Processor -- start 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.141+0000 7fb769331700 1 -- start start 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.141+0000 7fb769331700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb7640fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.141+0000 7fb769331700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7640fef40 con 0x7fb764070430 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.142+0000 7fb762ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb7640fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.142+0000 7fb762ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb7640fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41438/0 (socket says 172.21.15.104:41438) 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.142+0000 7fb762ffd700 1 -- 172.21.15.104:0/1988709897 learned_addr learned my addr 172.21.15.104:0/1988709897 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.142+0000 7fb762ffd700 1 -- 172.21.15.104:0/1988709897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7640ff080 con 0x7fb764070430 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.142+0000 7fb762ffd700 1 --2- 172.21.15.104:0/1988709897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb7640fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb75800b5f0 tx=0x7fb758006ee0).ready entity=mon.0 client_cookie=ccb4d02bb2cdd18a server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.142+0000 7fb761ffb700 1 -- 172.21.15.104:0/1988709897 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb75800d600 con 0x7fb764070430 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.142+0000 7fb761ffb700 1 -- 172.21.15.104:0/1988709897 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb75800dbf0 con 0x7fb764070430 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.142+0000 7fb761ffb700 1 -- 172.21.15.104:0/1988709897 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb758011cb0 con 0x7fb764070430 2024-02-12T16:28:58.191 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.143+0000 7fb769331700 1 -- 172.21.15.104:0/1988709897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 msgr2=0x7fb7640fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:58.194 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.143+0000 7fb769331700 1 --2- 172.21.15.104:0/1988709897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb7640fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb75800b5f0 tx=0x7fb758006ee0).stop 2024-02-12T16:28:58.194 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.143+0000 7fb769331700 1 -- 172.21.15.104:0/1988709897 shutdown_connections 2024-02-12T16:28:58.194 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.143+0000 7fb769331700 1 --2- 172.21.15.104:0/1988709897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb7640fea50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:58.194 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.143+0000 7fb769331700 1 -- 172.21.15.104:0/1988709897 >> 172.21.15.104:0/1988709897 conn(0x7fb7640f6530 msgr2=0x7fb7640f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.143+0000 7fb769331700 1 -- 172.21.15.104:0/1988709897 shutdown_connections 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.143+0000 7fb769331700 1 -- 172.21.15.104:0/1988709897 wait complete. 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.143+0000 7fb769331700 1 Processor -- start 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.143+0000 7fb769331700 1 -- start start 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.143+0000 7fb769331700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb76410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.143+0000 7fb769331700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76410b080 con 0x7fb764070430 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.144+0000 7fb762ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb76410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.144+0000 7fb762ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb76410ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41452/0 (socket says 172.21.15.104:41452) 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.144+0000 7fb762ffd700 1 -- 172.21.15.104:0/1400733055 learned_addr learned my addr 172.21.15.104:0/1400733055 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.144+0000 7fb762ffd700 1 -- 172.21.15.104:0/1400733055 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75800b200 con 0x7fb764070430 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.144+0000 7fb762ffd700 1 --2- 172.21.15.104:0/1400733055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb76410ab90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb75800df90 tx=0x7fb758006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.144+0000 7fb74b7fe700 1 -- 172.21.15.104:0/1400733055 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb75801b410 con 0x7fb764070430 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.144+0000 7fb74b7fe700 1 -- 172.21.15.104:0/1400733055 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb75801ba00 con 0x7fb764070430 2024-02-12T16:28:58.195 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.144+0000 7fb769331700 1 -- 172.21.15.104:0/1400733055 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76410b280 con 0x7fb764070430 2024-02-12T16:28:58.196 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.144+0000 7fb74b7fe700 1 -- 172.21.15.104:0/1400733055 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb75801aa30 con 0x7fb764070430 2024-02-12T16:28:58.196 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.144+0000 7fb769331700 1 -- 172.21.15.104:0/1400733055 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76410b710 con 0x7fb764070430 2024-02-12T16:28:58.196 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.145+0000 7fb74b7fe700 1 -- 172.21.15.104:0/1400733055 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb75800f070 con 0x7fb764070430 2024-02-12T16:28:58.196 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.145+0000 7fb769331700 1 -- 172.21.15.104:0/1400733055 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb764045ad0 con 0x7fb764070430 2024-02-12T16:28:58.196 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.145+0000 7fb74b7fe700 1 --2- 172.21.15.104:0/1400733055 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fb75002fe10 0x7fb7500322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:58.196 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.145+0000 7fb74b7fe700 1 -- 172.21.15.104:0/1400733055 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb7580411a0 con 0x7fb764070430 2024-02-12T16:28:58.196 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.145+0000 7fb7627fc700 1 --2- 172.21.15.104:0/1400733055 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fb75002fe10 0x7fb7500322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:58.196 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.145+0000 7fb7627fc700 1 --2- 172.21.15.104:0/1400733055 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fb75002fe10 0x7fb7500322d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb7540096a0 tx=0x7fb754006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:58.196 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.149+0000 7fb74b7fe700 1 -- 172.21.15.104:0/1400733055 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb75801ab90 con 0x7fb764070430 2024-02-12T16:28:58.329 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.326+0000 7fb769331700 1 -- 172.21.15.104:0/1400733055 --> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fb76410c030 con 0x7fb75002fe10 2024-02-12T16:28:58.332 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.330+0000 7fb74b7fe700 1 -- 172.21.15.104:0/1400733055 <== mgr.14114 v2:172.21.15.104:6800/3228071818 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb76410c030 con 0x7fb75002fe10 2024-02-12T16:28:58.333 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-12T16:28:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.332+0000 7fb769331700 1 -- 172.21.15.104:0/1400733055 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fb75002fe10 msgr2=0x7fb7500322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.332+0000 7fb769331700 1 --2- 172.21.15.104:0/1400733055 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fb75002fe10 0x7fb7500322d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb7540096a0 tx=0x7fb754006b40).stop 2024-02-12T16:28:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.332+0000 7fb769331700 1 -- 172.21.15.104:0/1400733055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 msgr2=0x7fb76410ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:58.335 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.332+0000 7fb769331700 1 --2- 172.21.15.104:0/1400733055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb76410ab90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb75800df90 tx=0x7fb758006ee0).stop 2024-02-12T16:28:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.332+0000 7fb769331700 1 -- 172.21.15.104:0/1400733055 shutdown_connections 2024-02-12T16:28:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.332+0000 7fb769331700 1 --2- 172.21.15.104:0/1400733055 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7fb75002fe10 0x7fb7500322d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.332+0000 7fb769331700 1 --2- 172.21.15.104:0/1400733055 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb764070430 0x7fb76410ab90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.332+0000 7fb769331700 1 -- 172.21.15.104:0/1400733055 >> 172.21.15.104:0/1400733055 conn(0x7fb7640f6530 msgr2=0x7fb7640f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.332+0000 7fb769331700 1 -- 172.21.15.104:0/1400733055 shutdown_connections 2024-02-12T16:28:58.336 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:58.332+0000 7fb769331700 1 -- 172.21.15.104:0/1400733055 wait complete. 2024-02-12T16:28:58.354 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:28:57.020019+0000 mgr.smithi104.ohwbhf (mgr.14114) 9 : cephadm [INF] Added host smithi104 2024-02-12T16:28:58.953 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying mgr service with default placement... 2024-02-12T16:28:58.954 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-12T16:28:59.474 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.471+0000 7f6736b92700 1 Processor -- start 2024-02-12T16:28:59.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.471+0000 7f6736b92700 1 -- start start 2024-02-12T16:28:59.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.472+0000 7f6736b92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:59.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.472+0000 7f6736b92700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67300fef80 con 0x7f67300703f0 2024-02-12T16:28:59.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.472+0000 7f673492e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:59.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.472+0000 7f673492e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41464/0 (socket says 172.21.15.104:41464) 2024-02-12T16:28:59.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.472+0000 7f673492e700 1 -- 172.21.15.104:0/3602052163 learned_addr learned my addr 172.21.15.104:0/3602052163 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:59.478 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.472+0000 7f673492e700 1 -- 172.21.15.104:0/3602052163 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67300ff0c0 con 0x7f67300703f0 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.472+0000 7f673492e700 1 --2- 172.21.15.104:0/3602052163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fea90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f672400b5f0 tx=0x7f6724006ee0).ready entity=mon.0 client_cookie=d958abd63393def0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.472+0000 7f672f7fe700 1 -- 172.21.15.104:0/3602052163 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672400d600 con 0x7f67300703f0 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.472+0000 7f672f7fe700 1 -- 172.21.15.104:0/3602052163 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f672400dbf0 con 0x7f67300703f0 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.472+0000 7f672f7fe700 1 -- 172.21.15.104:0/3602052163 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6724011ce0 con 0x7f67300703f0 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.473+0000 7f6736b92700 1 -- 172.21.15.104:0/3602052163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 msgr2=0x7f67300fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.473+0000 7f6736b92700 1 --2- 172.21.15.104:0/3602052163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fea90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f672400b5f0 tx=0x7f6724006ee0).stop 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.473+0000 7f6736b92700 1 -- 172.21.15.104:0/3602052163 shutdown_connections 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.473+0000 7f6736b92700 1 --2- 172.21.15.104:0/3602052163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fea90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.473+0000 7f6736b92700 1 -- 172.21.15.104:0/3602052163 >> 172.21.15.104:0/3602052163 conn(0x7f67300f6530 msgr2=0x7f67300f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.473+0000 7f6736b92700 1 -- 172.21.15.104:0/3602052163 shutdown_connections 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.473+0000 7f6736b92700 1 -- 172.21.15.104:0/3602052163 wait complete. 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.473+0000 7f6736b92700 1 Processor -- start 2024-02-12T16:28:59.479 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.473+0000 7f6736b92700 1 -- start start 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.474+0000 7f6736b92700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.474+0000 7f6736b92700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67300fa990 con 0x7f67300703f0 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.474+0000 7f673492e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.474+0000 7f673492e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41466/0 (socket says 172.21.15.104:41466) 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.474+0000 7f673492e700 1 -- 172.21.15.104:0/4075267038 learned_addr learned my addr 172.21.15.104:0/4075267038 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.474+0000 7f673492e700 1 -- 172.21.15.104:0/4075267038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672400b200 con 0x7f67300703f0 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.474+0000 7f673492e700 1 --2- 172.21.15.104:0/4075267038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fc510 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f67240115c0 tx=0x7f6724006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.474+0000 7f672d7fa700 1 -- 172.21.15.104:0/4075267038 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672401b410 con 0x7f67300703f0 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.474+0000 7f672d7fa700 1 -- 172.21.15.104:0/4075267038 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f672401ba00 con 0x7f67300703f0 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.474+0000 7f6736b92700 1 -- 172.21.15.104:0/4075267038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67300fab30 con 0x7f67300703f0 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.474+0000 7f672d7fa700 1 -- 172.21.15.104:0/4075267038 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672401aaa0 con 0x7f67300703f0 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.475+0000 7f6736b92700 1 -- 172.21.15.104:0/4075267038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67300fafc0 con 0x7f67300703f0 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.475+0000 7f672d7fa700 1 -- 172.21.15.104:0/4075267038 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f672400f070 con 0x7f67300703f0 2024-02-12T16:28:59.480 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.475+0000 7f672d7fa700 1 --2- 172.21.15.104:0/4075267038 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f671402fe70 0x7f6714032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:28:59.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.475+0000 7f672d7fa700 1 -- 172.21.15.104:0/4075267038 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6724041ff0 con 0x7f67300703f0 2024-02-12T16:28:59.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.475+0000 7f672ffff700 1 --2- 172.21.15.104:0/4075267038 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f671402fe70 0x7f6714032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:28:59.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.476+0000 7f6736b92700 1 -- 172.21.15.104:0/4075267038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6730059100 con 0x7f67300703f0 2024-02-12T16:28:59.481 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.476+0000 7f672ffff700 1 --2- 172.21.15.104:0/4075267038 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f671402fe70 0x7f6714032330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f671c0096a0 tx=0x7f671c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:28:59.482 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.480+0000 7f672d7fa700 1 -- 172.21.15.104:0/4075267038 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6724010be0 con 0x7f67300703f0 2024-02-12T16:28:59.658 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.655+0000 7f6736b92700 1 -- 172.21.15.104:0/4075267038 --> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f6730104b00 con 0x7f671402fe70 2024-02-12T16:28:59.661 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.659+0000 7f672d7fa700 1 -- 172.21.15.104:0/4075267038 <== mgr.14114 v2:172.21.15.104:6800/3228071818 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6730104b00 con 0x7f671402fe70 2024-02-12T16:28:59.661 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-12T16:28:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.661+0000 7f6736b92700 1 -- 172.21.15.104:0/4075267038 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f671402fe70 msgr2=0x7f6714032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.661+0000 7f6736b92700 1 --2- 172.21.15.104:0/4075267038 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f671402fe70 0x7f6714032330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f671c0096a0 tx=0x7f671c006b40).stop 2024-02-12T16:28:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.661+0000 7f6736b92700 1 -- 172.21.15.104:0/4075267038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 msgr2=0x7f67300fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:28:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.661+0000 7f6736b92700 1 --2- 172.21.15.104:0/4075267038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fc510 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f67240115c0 tx=0x7f6724006ee0).stop 2024-02-12T16:28:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.661+0000 7f6736b92700 1 -- 172.21.15.104:0/4075267038 shutdown_connections 2024-02-12T16:28:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.661+0000 7f6736b92700 1 --2- 172.21.15.104:0/4075267038 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f671402fe70 0x7f6714032330 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.661+0000 7f6736b92700 1 --2- 172.21.15.104:0/4075267038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67300703f0 0x7f67300fc510 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:28:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.661+0000 7f6736b92700 1 -- 172.21.15.104:0/4075267038 >> 172.21.15.104:0/4075267038 conn(0x7f67300f6530 msgr2=0x7f67300f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:28:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.661+0000 7f6736b92700 1 -- 172.21.15.104:0/4075267038 shutdown_connections 2024-02-12T16:28:59.664 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:28:59.661+0000 7f6736b92700 1 -- 172.21.15.104:0/4075267038 wait complete. 2024-02-12T16:28:59.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:59 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:58.328683+0000 mgr.smithi104.ohwbhf (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:28:59.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:59 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:28:58.329369+0000 mgr.smithi104.ohwbhf (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-12T16:28:59.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:28:59 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:58.331528+0000 mon.smithi104 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:00.210 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying crash service with default placement... 2024-02-12T16:29:00.210 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-12T16:29:00.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:59.657461+0000 mgr.smithi104.ohwbhf (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:29:00.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:28:59.657955+0000 mgr.smithi104.ohwbhf (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-12T16:29:00.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:28:59.660206+0000 mon.smithi104 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:00.784 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.781+0000 7f24bd9be700 1 Processor -- start 2024-02-12T16:29:00.784 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.781+0000 7f24bd9be700 1 -- start start 2024-02-12T16:29:00.784 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.781+0000 7f24bd9be700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b80fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:00.784 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.781+0000 7f24bd9be700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b80fc7b0 con 0x7f24b80fbea0 2024-02-12T16:29:00.784 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.781+0000 7f24b6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b80fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:00.784 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.781+0000 7f24b6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b80fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41472/0 (socket says 172.21.15.104:41472) 2024-02-12T16:29:00.784 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.781+0000 7f24b6ffd700 1 -- 172.21.15.104:0/2826212077 learned_addr learned my addr 172.21.15.104:0/2826212077 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:00.785 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.781+0000 7f24b6ffd700 1 -- 172.21.15.104:0/2826212077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b80fc8f0 con 0x7f24b80fbea0 2024-02-12T16:29:00.785 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.781+0000 7f24b6ffd700 1 --2- 172.21.15.104:0/2826212077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b80fc2c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f24a800b5f0 tx=0x7f24a8006ee0).ready entity=mon.0 client_cookie=ef423d190e6d3bc8 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:00.786 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.782+0000 7f24b5ffb700 1 -- 172.21.15.104:0/2826212077 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24a800d600 con 0x7f24b80fbea0 2024-02-12T16:29:00.786 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.782+0000 7f24b5ffb700 1 -- 172.21.15.104:0/2826212077 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f24a800dbf0 con 0x7f24b80fbea0 2024-02-12T16:29:00.786 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.782+0000 7f24b5ffb700 1 -- 172.21.15.104:0/2826212077 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24a8011cb0 con 0x7f24b80fbea0 2024-02-12T16:29:00.787 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.782+0000 7f24bd9be700 1 -- 172.21.15.104:0/2826212077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 msgr2=0x7f24b80fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:00.787 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.782+0000 7f24bd9be700 1 --2- 172.21.15.104:0/2826212077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b80fc2c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f24a800b5f0 tx=0x7f24a8006ee0).stop 2024-02-12T16:29:00.787 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.782+0000 7f24bd9be700 1 -- 172.21.15.104:0/2826212077 shutdown_connections 2024-02-12T16:29:00.787 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.782+0000 7f24bd9be700 1 --2- 172.21.15.104:0/2826212077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b80fc2c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:00.787 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.782+0000 7f24bd9be700 1 -- 172.21.15.104:0/2826212077 >> 172.21.15.104:0/2826212077 conn(0x7f24b80f76a0 msgr2=0x7f24b80f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:00.787 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.782+0000 7f24bd9be700 1 -- 172.21.15.104:0/2826212077 shutdown_connections 2024-02-12T16:29:00.787 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.782+0000 7f24bd9be700 1 -- 172.21.15.104:0/2826212077 wait complete. 2024-02-12T16:29:00.787 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.783+0000 7f24bd9be700 1 Processor -- start 2024-02-12T16:29:00.787 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.783+0000 7f24bd9be700 1 -- start start 2024-02-12T16:29:00.787 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.783+0000 7f24bd9be700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b810abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:00.787 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.783+0000 7f24bd9be700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b810b0b0 con 0x7f24b80fbea0 2024-02-12T16:29:00.788 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.783+0000 7f24b6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b810abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:00.788 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.783+0000 7f24b6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b810abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41478/0 (socket says 172.21.15.104:41478) 2024-02-12T16:29:00.788 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.783+0000 7f24b6ffd700 1 -- 172.21.15.104:0/3648573064 learned_addr learned my addr 172.21.15.104:0/3648573064 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:00.788 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.784+0000 7f24b6ffd700 1 -- 172.21.15.104:0/3648573064 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a800b200 con 0x7f24b80fbea0 2024-02-12T16:29:00.788 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.784+0000 7f24b6ffd700 1 --2- 172.21.15.104:0/3648573064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b810abc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f24a8002f40 tx=0x7f24a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:00.788 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.784+0000 7f249f7fe700 1 -- 172.21.15.104:0/3648573064 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24a801b410 con 0x7f24b80fbea0 2024-02-12T16:29:00.788 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.784+0000 7f249f7fe700 1 -- 172.21.15.104:0/3648573064 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f24a801ba00 con 0x7f24b80fbea0 2024-02-12T16:29:00.788 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.784+0000 7f24bd9be700 1 -- 172.21.15.104:0/3648573064 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b810b2b0 con 0x7f24b80fbea0 2024-02-12T16:29:00.788 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.784+0000 7f249f7fe700 1 -- 172.21.15.104:0/3648573064 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24a801a980 con 0x7f24b80fbea0 2024-02-12T16:29:00.788 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.784+0000 7f24bd9be700 1 -- 172.21.15.104:0/3648573064 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b810b740 con 0x7f24b80fbea0 2024-02-12T16:29:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.785+0000 7f24bd9be700 1 -- 172.21.15.104:0/3648573064 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b8045ad0 con 0x7f24b80fbea0 2024-02-12T16:29:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.785+0000 7f249f7fe700 1 -- 172.21.15.104:0/3648573064 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f24a800f070 con 0x7f24b80fbea0 2024-02-12T16:29:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.785+0000 7f249f7fe700 1 --2- 172.21.15.104:0/3648573064 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f24a402fe10 0x7f24a40322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.785+0000 7f249f7fe700 1 -- 172.21.15.104:0/3648573064 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f24a8041400 con 0x7f24b80fbea0 2024-02-12T16:29:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.785+0000 7f24b67fc700 1 --2- 172.21.15.104:0/3648573064 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f24a402fe10 0x7f24a40322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:00.789 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.786+0000 7f24b67fc700 1 --2- 172.21.15.104:0/3648573064 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f24a402fe10 0x7f24a40322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f24ac0096a0 tx=0x7f24ac006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:00.791 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.789+0000 7f249f7fe700 1 -- 172.21.15.104:0/3648573064 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24a8022310 con 0x7f24b80fbea0 2024-02-12T16:29:00.966 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.964+0000 7f24bd9be700 1 -- 172.21.15.104:0/3648573064 --> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f24b810c060 con 0x7f24a402fe10 2024-02-12T16:29:00.970 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.967+0000 7f249f7fe700 1 -- 172.21.15.104:0/3648573064 <== mgr.14114 v2:172.21.15.104:6800/3228071818 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f24b810c060 con 0x7f24a402fe10 2024-02-12T16:29:00.970 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-12T16:29:00.972 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.969+0000 7f24bd9be700 1 -- 172.21.15.104:0/3648573064 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f24a402fe10 msgr2=0x7f24a40322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:00.972 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.969+0000 7f24bd9be700 1 --2- 172.21.15.104:0/3648573064 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f24a402fe10 0x7f24a40322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f24ac0096a0 tx=0x7f24ac006b40).stop 2024-02-12T16:29:00.972 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.969+0000 7f24bd9be700 1 -- 172.21.15.104:0/3648573064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 msgr2=0x7f24b810abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:00.972 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.969+0000 7f24bd9be700 1 --2- 172.21.15.104:0/3648573064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b810abc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f24a8002f40 tx=0x7f24a8006ee0).stop 2024-02-12T16:29:00.972 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.969+0000 7f24bd9be700 1 -- 172.21.15.104:0/3648573064 shutdown_connections 2024-02-12T16:29:00.973 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.969+0000 7f24bd9be700 1 --2- 172.21.15.104:0/3648573064 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f24a402fe10 0x7f24a40322d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:00.973 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.969+0000 7f24bd9be700 1 --2- 172.21.15.104:0/3648573064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24b80fbea0 0x7f24b810abc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:00.973 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.969+0000 7f24bd9be700 1 -- 172.21.15.104:0/3648573064 >> 172.21.15.104:0/3648573064 conn(0x7f24b80f76a0 msgr2=0x7f24b80f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:00.973 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.969+0000 7f24bd9be700 1 -- 172.21.15.104:0/3648573064 shutdown_connections 2024-02-12T16:29:00.973 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:00.969+0000 7f24bd9be700 1 -- 172.21.15.104:0/3648573064 wait complete. 2024-02-12T16:29:01.532 INFO:teuthology.orchestra.run.smithi104.stderr:Enabling mgr prometheus module... 2024-02-12T16:29:01.533 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-12T16:29:02.134 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.129+0000 7f42f04f9700 1 Processor -- start 2024-02-12T16:29:02.134 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.129+0000 7f42f04f9700 1 -- start start 2024-02-12T16:29:02.134 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.130+0000 7f42f04f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e80fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:02.134 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.130+0000 7f42f04f9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e80fb6a0 con 0x7f42e80fad90 2024-02-12T16:29:02.134 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.130+0000 7f42ee295700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e80fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.130+0000 7f42ee295700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e80fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41492/0 (socket says 172.21.15.104:41492) 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.130+0000 7f42ee295700 1 -- 172.21.15.104:0/4207924705 learned_addr learned my addr 172.21.15.104:0/4207924705 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.130+0000 7f42ee295700 1 -- 172.21.15.104:0/4207924705 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e80fb7e0 con 0x7f42e80fad90 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.130+0000 7f42ee295700 1 --2- 172.21.15.104:0/4207924705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e80fb1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f42d800b5f0 tx=0x7f42d8006ee0).ready entity=mon.0 client_cookie=9d23bcc335ef07be server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.130+0000 7f42ed293700 1 -- 172.21.15.104:0/4207924705 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42d800d600 con 0x7f42e80fad90 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.130+0000 7f42ed293700 1 -- 172.21.15.104:0/4207924705 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f42d800dbf0 con 0x7f42e80fad90 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.130+0000 7f42ed293700 1 -- 172.21.15.104:0/4207924705 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42d8011cb0 con 0x7f42e80fad90 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.131+0000 7f42f04f9700 1 -- 172.21.15.104:0/4207924705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 msgr2=0x7f42e80fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.131+0000 7f42f04f9700 1 --2- 172.21.15.104:0/4207924705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e80fb1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f42d800b5f0 tx=0x7f42d8006ee0).stop 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.131+0000 7f42f04f9700 1 -- 172.21.15.104:0/4207924705 shutdown_connections 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.131+0000 7f42f04f9700 1 --2- 172.21.15.104:0/4207924705 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e80fb1b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.131+0000 7f42f04f9700 1 -- 172.21.15.104:0/4207924705 >> 172.21.15.104:0/4207924705 conn(0x7f42e80f6530 msgr2=0x7f42e80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.131+0000 7f42f04f9700 1 -- 172.21.15.104:0/4207924705 shutdown_connections 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.131+0000 7f42f04f9700 1 -- 172.21.15.104:0/4207924705 wait complete. 2024-02-12T16:29:02.135 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42f04f9700 1 Processor -- start 2024-02-12T16:29:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42f04f9700 1 -- start start 2024-02-12T16:29:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42f04f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e810ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42f04f9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e810b000 con 0x7f42e80fad90 2024-02-12T16:29:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42ee295700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e810ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42ee295700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e810ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:41502/0 (socket says 172.21.15.104:41502) 2024-02-12T16:29:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42ee295700 1 -- 172.21.15.104:0/3111967027 learned_addr learned my addr 172.21.15.104:0/3111967027 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:02.136 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42ee295700 1 -- 172.21.15.104:0/3111967027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d800b200 con 0x7f42e80fad90 2024-02-12T16:29:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42ee295700 1 --2- 172.21.15.104:0/3111967027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e810ab10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f42d8011ea0 tx=0x7f42d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42d6ffd700 1 -- 172.21.15.104:0/3111967027 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42d801b570 con 0x7f42e80fad90 2024-02-12T16:29:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42d6ffd700 1 -- 172.21.15.104:0/3111967027 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f42d801bb60 con 0x7f42e80fad90 2024-02-12T16:29:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42f04f9700 1 -- 172.21.15.104:0/3111967027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e810b200 con 0x7f42e80fad90 2024-02-12T16:29:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42d6ffd700 1 -- 172.21.15.104:0/3111967027 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42d801a8d0 con 0x7f42e80fad90 2024-02-12T16:29:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.132+0000 7f42f04f9700 1 -- 172.21.15.104:0/3111967027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e810b690 con 0x7f42e80fad90 2024-02-12T16:29:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.133+0000 7f42d6ffd700 1 -- 172.21.15.104:0/3111967027 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f42d800f070 con 0x7f42e80fad90 2024-02-12T16:29:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.133+0000 7f42f04f9700 1 -- 172.21.15.104:0/3111967027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e8045ad0 con 0x7f42e80fad90 2024-02-12T16:29:02.137 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.133+0000 7f42d6ffd700 1 --2- 172.21.15.104:0/3111967027 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f42d002fe10 0x7f42d00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:02.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.133+0000 7f42d6ffd700 1 -- 172.21.15.104:0/3111967027 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f42d8041470 con 0x7f42e80fad90 2024-02-12T16:29:02.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.134+0000 7f42eda94700 1 --2- 172.21.15.104:0/3111967027 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f42d002fe10 0x7f42d00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:02.138 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.134+0000 7f42eda94700 1 --2- 172.21.15.104:0/3111967027 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f42d002fe10 0x7f42d00322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f42e40096a0 tx=0x7f42e4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:02.140 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.138+0000 7f42d6ffd700 1 -- 172.21.15.104:0/3111967027 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42d80224f0 con 0x7f42e80fad90 2024-02-12T16:29:02.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:00.965819+0000 mgr.smithi104.ohwbhf (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:29:02.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:29:00.966479+0000 mgr.smithi104.ohwbhf (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-12T16:29:02.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:00.968667+0000 mon.smithi104 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.104:0/2959685700' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:02.325 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:02.323+0000 7f42f04f9700 1 -- 172.21.15.104:0/3111967027 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f42e80fcae0 con 0x7f42e80fad90 2024-02-12T16:29:03.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.033+0000 7f42d6ffd700 1 -- 172.21.15.104:0/3111967027 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f42d8022d80 con 0x7f42e80fad90 2024-02-12T16:29:03.036 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.034+0000 7f42d6ffd700 1 -- 172.21.15.104:0/3111967027 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f42d80194e0 con 0x7f42e80fad90 2024-02-12T16:29:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.036+0000 7f42f04f9700 1 -- 172.21.15.104:0/3111967027 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f42d002fe10 msgr2=0x7f42d00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.036+0000 7f42f04f9700 1 --2- 172.21.15.104:0/3111967027 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f42d002fe10 0x7f42d00322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f42e40096a0 tx=0x7f42e4006b40).stop 2024-02-12T16:29:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.036+0000 7f42f04f9700 1 -- 172.21.15.104:0/3111967027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 msgr2=0x7f42e810ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.036+0000 7f42f04f9700 1 --2- 172.21.15.104:0/3111967027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e810ab10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f42d8011ea0 tx=0x7f42d8006ee0).stop 2024-02-12T16:29:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.037+0000 7f42f04f9700 1 -- 172.21.15.104:0/3111967027 shutdown_connections 2024-02-12T16:29:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.037+0000 7f42f04f9700 1 --2- 172.21.15.104:0/3111967027 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f42d002fe10 0x7f42d00322d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.037+0000 7f42f04f9700 1 --2- 172.21.15.104:0/3111967027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42e80fad90 0x7f42e810ab10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.037+0000 7f42f04f9700 1 -- 172.21.15.104:0/3111967027 >> 172.21.15.104:0/3111967027 conn(0x7f42e80f6530 msgr2=0x7f42e80f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.037+0000 7f42f04f9700 1 -- 172.21.15.104:0/3111967027 shutdown_connections 2024-02-12T16:29:03.040 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:03.037+0000 7f42f04f9700 1 -- 172.21.15.104:0/3111967027 wait complete. 2024-02-12T16:29:03.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:03 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:02.324500+0000 mon.smithi104 (mon.0) 59 : audit [INF] from='client.? 172.21.15.104:0/3111967027' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-12T16:29:03.714 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying prometheus service with default placement... 2024-02-12T16:29:03.715 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-12T16:29:04.394 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.355+0000 7f8d6d272700 1 Processor -- start 2024-02-12T16:29:04.395 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.355+0000 7f8d6d272700 1 -- start start 2024-02-12T16:29:04.395 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.355+0000 7f8d6d272700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d680fea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:04.395 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.355+0000 7f8d6d272700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d68070770 con 0x7f8d68070350 2024-02-12T16:29:04.395 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.355+0000 7f8d67fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d680fea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:04.395 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.355+0000 7f8d67fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d680fea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34134/0 (socket says 172.21.15.104:34134) 2024-02-12T16:29:04.395 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.355+0000 7f8d67fff700 1 -- 172.21.15.104:0/550079850 learned_addr learned my addr 172.21.15.104:0/550079850 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:04.395 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.355+0000 7f8d67fff700 1 -- 172.21.15.104:0/550079850 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d680fef20 con 0x7f8d68070350 2024-02-12T16:29:04.395 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.356+0000 7f8d67fff700 1 --2- 172.21.15.104:0/550079850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d680fea30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f8d5800b5f0 tx=0x7f8d58006ee0).ready entity=mon.0 client_cookie=7812f449687dfd28 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:04.395 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.356+0000 7f8d66ffd700 1 -- 172.21.15.104:0/550079850 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d5800d600 con 0x7f8d68070350 2024-02-12T16:29:04.395 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.356+0000 7f8d66ffd700 1 -- 172.21.15.104:0/550079850 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8d5800dbf0 con 0x7f8d68070350 2024-02-12T16:29:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.356+0000 7f8d66ffd700 1 -- 172.21.15.104:0/550079850 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d58011cb0 con 0x7f8d68070350 2024-02-12T16:29:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.356+0000 7f8d6d272700 1 -- 172.21.15.104:0/550079850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 msgr2=0x7f8d680fea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.356+0000 7f8d6d272700 1 --2- 172.21.15.104:0/550079850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d680fea30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f8d5800b5f0 tx=0x7f8d58006ee0).stop 2024-02-12T16:29:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.356+0000 7f8d6d272700 1 -- 172.21.15.104:0/550079850 shutdown_connections 2024-02-12T16:29:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.356+0000 7f8d6d272700 1 --2- 172.21.15.104:0/550079850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d680fea30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.356+0000 7f8d6d272700 1 -- 172.21.15.104:0/550079850 >> 172.21.15.104:0/550079850 conn(0x7f8d680f6410 msgr2=0x7f8d680f8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.356+0000 7f8d6d272700 1 -- 172.21.15.104:0/550079850 shutdown_connections 2024-02-12T16:29:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.356+0000 7f8d6d272700 1 -- 172.21.15.104:0/550079850 wait complete. 2024-02-12T16:29:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.357+0000 7f8d6d272700 1 Processor -- start 2024-02-12T16:29:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.357+0000 7f8d6d272700 1 -- start start 2024-02-12T16:29:04.396 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.357+0000 7f8d6d272700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d6810a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.357+0000 7f8d6d272700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6810ae90 con 0x7f8d68070350 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.357+0000 7f8d67fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d6810a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.357+0000 7f8d67fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d6810a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34140/0 (socket says 172.21.15.104:34140) 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.357+0000 7f8d67fff700 1 -- 172.21.15.104:0/1532811642 learned_addr learned my addr 172.21.15.104:0/1532811642 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.358+0000 7f8d67fff700 1 -- 172.21.15.104:0/1532811642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5800b200 con 0x7f8d68070350 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.358+0000 7f8d67fff700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d6810a9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f8d5800df90 tx=0x7f8d58006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.358+0000 7f8d64ff9700 1 -- 172.21.15.104:0/1532811642 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d5801b410 con 0x7f8d68070350 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.358+0000 7f8d64ff9700 1 -- 172.21.15.104:0/1532811642 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8d5801ba00 con 0x7f8d68070350 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.358+0000 7f8d6d272700 1 -- 172.21.15.104:0/1532811642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6810b090 con 0x7f8d68070350 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.358+0000 7f8d64ff9700 1 -- 172.21.15.104:0/1532811642 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d5801aa30 con 0x7f8d68070350 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.358+0000 7f8d6d272700 1 -- 172.21.15.104:0/1532811642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6810b520 con 0x7f8d68070350 2024-02-12T16:29:04.397 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.359+0000 7f8d6d272700 1 -- 172.21.15.104:0/1532811642 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d68045ad0 con 0x7f8d68070350 2024-02-12T16:29:04.398 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.359+0000 7f8d64ff9700 1 -- 172.21.15.104:0/1532811642 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f8d5800f070 con 0x7f8d68070350 2024-02-12T16:29:04.398 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.359+0000 7f8d64ff9700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 0x7f8d54032380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:04.398 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.359+0000 7f8d64ff9700 1 -- 172.21.15.104:0/1532811642 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8d580411d0 con 0x7f8d68070350 2024-02-12T16:29:04.398 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.359+0000 7f8d677fe700 1 -- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 msgr2=0x7f8d54032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/3228071818 2024-02-12T16:29:04.398 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.359+0000 7f8d677fe700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 0x7f8d54032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-12T16:29:04.398 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.366+0000 7f8d64ff9700 1 -- 172.21.15.104:0/1532811642 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d58022310 con 0x7f8d68070350 2024-02-12T16:29:04.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:03.035582+0000 mon.smithi104 (mon.0) 60 : audit [INF] from='client.? 172.21.15.104:0/3111967027' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-12T16:29:04.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:03.035628+0000 mon.smithi104 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi104.ohwbhf(active, since 19s) 2024-02-12T16:29:04.555 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.553+0000 7f8d6d272700 1 -- 172.21.15.104:0/1532811642 --> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d6810bd80 con 0x7f8d5402fec0 2024-02-12T16:29:04.562 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.560+0000 7f8d677fe700 1 -- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 msgr2=0x7f8d54032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/3228071818 2024-02-12T16:29:04.562 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.560+0000 7f8d677fe700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 0x7f8d54032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-12T16:29:04.963 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.961+0000 7f8d677fe700 1 -- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 msgr2=0x7f8d54032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/3228071818 2024-02-12T16:29:04.963 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:04.961+0000 7f8d677fe700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 0x7f8d54032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-12T16:29:05.764 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:05.762+0000 7f8d677fe700 1 -- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 msgr2=0x7f8d54032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/3228071818 2024-02-12T16:29:05.764 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:05.762+0000 7f8d677fe700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 0x7f8d54032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-12T16:29:07.366 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:07.363+0000 7f8d677fe700 1 -- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 msgr2=0x7f8d54032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/3228071818 2024-02-12T16:29:07.367 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:07.363+0000 7f8d677fe700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 0x7f8d54032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-12T16:29:08.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:08.032+0000 7f8d64ff9700 1 -- 172.21.15.104:0/1532811642 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f8d5801ab90 con 0x7f8d68070350 2024-02-12T16:29:08.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:08.032+0000 7f8d64ff9700 1 -- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 msgr2=0x7f8d54032380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-12T16:29:08.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:08.032+0000 7f8d64ff9700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 0x7f8d54032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-12T16:29:08.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:07.979000+0000 mon.smithi104 (mon.0) 62 : cluster [INF] Active manager daemon smithi104.ohwbhf restarted 2024-02-12T16:29:08.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:07.979395+0000 mon.smithi104 (mon.0) 63 : cluster [INF] Activating manager daemon smithi104.ohwbhf 2024-02-12T16:29:08.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:07.981200+0000 mon.smithi104 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-12T16:29:09.037 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.033+0000 7f8d64ff9700 1 -- 172.21.15.104:0/1532811642 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f8d58022d70 con 0x7f8d68070350 2024-02-12T16:29:09.037 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.033+0000 7f8d64ff9700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f8d54033ce0 0x7f8d540360d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:09.037 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.034+0000 7f8d64ff9700 1 -- 172.21.15.104:0/1532811642 --> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d6810bd80 con 0x7f8d54033ce0 2024-02-12T16:29:09.037 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.034+0000 7f8d677fe700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f8d54033ce0 0x7f8d540360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:09.037 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.034+0000 7f8d677fe700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f8d54033ce0 0x7f8d540360d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8d5c006fd0 tx=0x7f8d5c006d30).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.040+0000 7f8d64ff9700 1 -- 172.21.15.104:0/1532811642 <== mgr.14142 v2:172.21.15.104:6800/3327302611 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f8d6810bd80 con 0x7f8d54033ce0 2024-02-12T16:29:09.043 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-12T16:29:09.045 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.042+0000 7f8d6d272700 1 -- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f8d54033ce0 msgr2=0x7f8d540360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:09.045 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.042+0000 7f8d6d272700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f8d54033ce0 0x7f8d540360d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8d5c006fd0 tx=0x7f8d5c006d30).stop 2024-02-12T16:29:09.046 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.042+0000 7f8d6d272700 1 -- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 msgr2=0x7f8d6810a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:09.046 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.042+0000 7f8d6d272700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d6810a9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f8d5800df90 tx=0x7f8d58006ee0).stop 2024-02-12T16:29:09.046 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.042+0000 7f8d6d272700 1 -- 172.21.15.104:0/1532811642 shutdown_connections 2024-02-12T16:29:09.046 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.042+0000 7f8d6d272700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f8d54033ce0 0x7f8d540360d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:09.046 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.042+0000 7f8d6d272700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8d68070350 0x7f8d6810a9a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:09.046 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.042+0000 7f8d6d272700 1 --2- 172.21.15.104:0/1532811642 >> [v2:172.21.15.104:6800/3228071818,v1:172.21.15.104:6801/3228071818] conn(0x7f8d5402fec0 0x7f8d54032380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-12T16:29:09.046 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.042+0000 7f8d6d272700 1 -- 172.21.15.104:0/1532811642 >> 172.21.15.104:0/1532811642 conn(0x7f8d680f6410 msgr2=0x7f8d680f7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:09.046 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.042+0000 7f8d6d272700 1 -- 172.21.15.104:0/1532811642 shutdown_connections 2024-02-12T16:29:09.046 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:09.042+0000 7f8d6d272700 1 -- 172.21.15.104:0/1532811642 wait complete. 2024-02-12T16:29:09.402 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:08.033958+0000 mon.smithi104 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi104.ohwbhf(active, starting, since 0.0546426s) 2024-02-12T16:29:09.402 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:08.034960+0000 mon.smithi104 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-12T16:29:09.402 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:08.035506+0000 mon.smithi104 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mgr metadata", "who": "smithi104.ohwbhf", "id": "smithi104.ohwbhf"}]: dispatch 2024-02-12T16:29:09.403 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:08.036123+0000 mon.smithi104 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T16:29:09.403 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:08.036269+0000 mon.smithi104 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T16:29:09.403 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:08.036374+0000 mon.smithi104 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T16:29:09.403 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:08.049575+0000 mon.smithi104 (mon.0) 71 : cluster [INF] Manager daemon smithi104.ohwbhf is now available 2024-02-12T16:29:09.403 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:08.146019+0000 mon.smithi104 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:29:09.403 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:08.147517+0000 mon.smithi104 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:29:09.403 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:08.167569+0000 mon.smithi104 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:29:09.403 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:08.170997+0000 mon.smithi104 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:29:09.672 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying grafana service with default placement... 2024-02-12T16:29:09.672 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-12T16:29:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.236+0000 7f447b06e700 1 Processor -- start 2024-02-12T16:29:10.241 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.236+0000 7f447b06e700 1 -- start start 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.236+0000 7f447b06e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f44740fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.236+0000 7f447b06e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44740fb6a0 con 0x7f44740fad90 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.237+0000 7f4478e0a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f44740fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.237+0000 7f4478e0a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f44740fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34198/0 (socket says 172.21.15.104:34198) 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.237+0000 7f4478e0a700 1 -- 172.21.15.104:0/2453646338 learned_addr learned my addr 172.21.15.104:0/2453646338 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.237+0000 7f4478e0a700 1 -- 172.21.15.104:0/2453646338 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44740fb7e0 con 0x7f44740fad90 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.237+0000 7f4478e0a700 1 --2- 172.21.15.104:0/2453646338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f44740fb1b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f446800b5f0 tx=0x7f4468006ee0).ready entity=mon.0 client_cookie=be1f2faaea6b18a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.237+0000 7f44737fe700 1 -- 172.21.15.104:0/2453646338 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f446800d600 con 0x7f44740fad90 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.237+0000 7f44737fe700 1 -- 172.21.15.104:0/2453646338 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f446800dbf0 con 0x7f44740fad90 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.237+0000 7f44737fe700 1 -- 172.21.15.104:0/2453646338 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4468011cb0 con 0x7f44740fad90 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.238+0000 7f447b06e700 1 -- 172.21.15.104:0/2453646338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 msgr2=0x7f44740fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.238+0000 7f447b06e700 1 --2- 172.21.15.104:0/2453646338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f44740fb1b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f446800b5f0 tx=0x7f4468006ee0).stop 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.238+0000 7f447b06e700 1 -- 172.21.15.104:0/2453646338 shutdown_connections 2024-02-12T16:29:10.242 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.238+0000 7f447b06e700 1 --2- 172.21.15.104:0/2453646338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f44740fb1b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.238+0000 7f447b06e700 1 -- 172.21.15.104:0/2453646338 >> 172.21.15.104:0/2453646338 conn(0x7f44740f6530 msgr2=0x7f44740f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.238+0000 7f447b06e700 1 -- 172.21.15.104:0/2453646338 shutdown_connections 2024-02-12T16:29:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.238+0000 7f447b06e700 1 -- 172.21.15.104:0/2453646338 wait complete. 2024-02-12T16:29:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.238+0000 7f447b06e700 1 Processor -- start 2024-02-12T16:29:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.238+0000 7f447b06e700 1 -- start start 2024-02-12T16:29:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f447b06e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f447410ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f447b06e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447410d300 con 0x7f44740fad90 2024-02-12T16:29:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f4478e0a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f447410ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f4478e0a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f447410ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34202/0 (socket says 172.21.15.104:34202) 2024-02-12T16:29:10.243 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f4478e0a700 1 -- 172.21.15.104:0/522958381 learned_addr learned my addr 172.21.15.104:0/522958381 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f4478e0a700 1 -- 172.21.15.104:0/522958381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446800b200 con 0x7f44740fad90 2024-02-12T16:29:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f4478e0a700 1 --2- 172.21.15.104:0/522958381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f447410ce10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f446800bf40 tx=0x7f4468006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f44717fa700 1 -- 172.21.15.104:0/522958381 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f446801b570 con 0x7f44740fad90 2024-02-12T16:29:10.244 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f447b06e700 1 -- 172.21.15.104:0/522958381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447410d500 con 0x7f44740fad90 2024-02-12T16:29:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f44717fa700 1 -- 172.21.15.104:0/522958381 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f446801bb60 con 0x7f44740fad90 2024-02-12T16:29:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f44717fa700 1 -- 172.21.15.104:0/522958381 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f446801aa20 con 0x7f44740fad90 2024-02-12T16:29:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.239+0000 7f447b06e700 1 -- 172.21.15.104:0/522958381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447410e110 con 0x7f44740fad90 2024-02-12T16:29:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.240+0000 7f44717fa700 1 -- 172.21.15.104:0/522958381 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f446800f070 con 0x7f44740fad90 2024-02-12T16:29:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.240+0000 7f44717fa700 1 --2- 172.21.15.104:0/522958381 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f445802ff20 0x7f44580323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.240+0000 7f44717fa700 1 -- 172.21.15.104:0/522958381 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4468041660 con 0x7f44740fad90 2024-02-12T16:29:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.240+0000 7f447b06e700 1 -- 172.21.15.104:0/522958381 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4474059100 con 0x7f44740fad90 2024-02-12T16:29:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.240+0000 7f4473fff700 1 --2- 172.21.15.104:0/522958381 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f445802ff20 0x7f44580323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:10.245 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.241+0000 7f4473fff700 1 --2- 172.21.15.104:0/522958381 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f445802ff20 0x7f44580323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f44600096a0 tx=0x7f4460006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:10.247 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.245+0000 7f44717fa700 1 -- 172.21.15.104:0/522958381 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4468016070 con 0x7f44740fad90 2024-02-12T16:29:10.422 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.419+0000 7f447b06e700 1 -- 172.21.15.104:0/522958381 --> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f44740fcbf0 con 0x7f445802ff20 2024-02-12T16:29:10.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:09.036373+0000 mon.smithi104 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi104.ohwbhf(active, since 1.05706s) 2024-02-12T16:29:10.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:29:09.037888+0000 mgr.smithi104.ohwbhf (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-12T16:29:10.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:09.041485+0000 mon.smithi104 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:10.428 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.426+0000 7f44717fa700 1 -- 172.21.15.104:0/522958381 <== mgr.14142 v2:172.21.15.104:6800/3327302611 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f44740fcbf0 con 0x7f445802ff20 2024-02-12T16:29:10.428 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-12T16:29:10.431 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.428+0000 7f447b06e700 1 -- 172.21.15.104:0/522958381 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f445802ff20 msgr2=0x7f44580323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:10.431 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.428+0000 7f447b06e700 1 --2- 172.21.15.104:0/522958381 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f445802ff20 0x7f44580323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f44600096a0 tx=0x7f4460006b40).stop 2024-02-12T16:29:10.431 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.428+0000 7f447b06e700 1 -- 172.21.15.104:0/522958381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 msgr2=0x7f447410ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:10.431 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.428+0000 7f447b06e700 1 --2- 172.21.15.104:0/522958381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f447410ce10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f446800bf40 tx=0x7f4468006ee0).stop 2024-02-12T16:29:10.431 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.428+0000 7f447b06e700 1 -- 172.21.15.104:0/522958381 shutdown_connections 2024-02-12T16:29:10.431 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.428+0000 7f447b06e700 1 --2- 172.21.15.104:0/522958381 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f445802ff20 0x7f44580323e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:10.431 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.428+0000 7f447b06e700 1 --2- 172.21.15.104:0/522958381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44740fad90 0x7f447410ce10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:10.432 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.428+0000 7f447b06e700 1 -- 172.21.15.104:0/522958381 >> 172.21.15.104:0/522958381 conn(0x7f44740f6530 msgr2=0x7f4474070070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:10.432 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.428+0000 7f447b06e700 1 -- 172.21.15.104:0/522958381 shutdown_connections 2024-02-12T16:29:10.432 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:10.428+0000 7f447b06e700 1 -- 172.21.15.104:0/522958381 wait complete. 2024-02-12T16:29:11.123 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying node-exporter service with default placement... 2024-02-12T16:29:11.123 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-12T16:29:11.125 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:10.045589+0000 mon.smithi104 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi104.ohwbhf(active, since 2s) 2024-02-12T16:29:11.125 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:10.427424+0000 mon.smithi104 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:11.651 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.646+0000 7fb9faa4d700 1 Processor -- start 2024-02-12T16:29:11.651 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.646+0000 7fb9faa4d700 1 -- start start 2024-02-12T16:29:11.651 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.646+0000 7fb9faa4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f4100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:11.651 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.646+0000 7fb9faa4d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f40fc6c0 con 0x7fb9f40703f0 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.646+0000 7fb9f9a4b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f4100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.646+0000 7fb9f9a4b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f4100cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34210/0 (socket says 172.21.15.104:34210) 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.646+0000 7fb9f9a4b700 1 -- 172.21.15.104:0/3052943571 learned_addr learned my addr 172.21.15.104:0/3052943571 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.646+0000 7fb9f9a4b700 1 -- 172.21.15.104:0/3052943571 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f41011c0 con 0x7fb9f40703f0 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.647+0000 7fb9f9a4b700 1 --2- 172.21.15.104:0/3052943571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f4100cd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb9f000b5f0 tx=0x7fb9f0006ee0).ready entity=mon.0 client_cookie=bbb174020fac62f server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.647+0000 7fb9f8a49700 1 -- 172.21.15.104:0/3052943571 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9f000d600 con 0x7fb9f40703f0 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.647+0000 7fb9f8a49700 1 -- 172.21.15.104:0/3052943571 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb9f000dbf0 con 0x7fb9f40703f0 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.647+0000 7fb9f8a49700 1 -- 172.21.15.104:0/3052943571 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9f0011ce0 con 0x7fb9f40703f0 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.647+0000 7fb9faa4d700 1 -- 172.21.15.104:0/3052943571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 msgr2=0x7fb9f4100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.647+0000 7fb9faa4d700 1 --2- 172.21.15.104:0/3052943571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f4100cd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb9f000b5f0 tx=0x7fb9f0006ee0).stop 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.647+0000 7fb9faa4d700 1 -- 172.21.15.104:0/3052943571 shutdown_connections 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.647+0000 7fb9faa4d700 1 --2- 172.21.15.104:0/3052943571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f4100cd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:11.652 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.647+0000 7fb9faa4d700 1 -- 172.21.15.104:0/3052943571 >> 172.21.15.104:0/3052943571 conn(0x7fb9f40f6390 msgr2=0x7fb9f40f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.647+0000 7fb9faa4d700 1 -- 172.21.15.104:0/3052943571 shutdown_connections 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.647+0000 7fb9faa4d700 1 -- 172.21.15.104:0/3052943571 wait complete. 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.648+0000 7fb9faa4d700 1 Processor -- start 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.648+0000 7fb9faa4d700 1 -- start start 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.648+0000 7fb9faa4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f410cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.648+0000 7fb9faa4d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f410d070 con 0x7fb9f40703f0 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.648+0000 7fb9f9a4b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f410cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.648+0000 7fb9f9a4b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f410cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34226/0 (socket says 172.21.15.104:34226) 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.648+0000 7fb9f9a4b700 1 -- 172.21.15.104:0/2737248093 learned_addr learned my addr 172.21.15.104:0/2737248093 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.648+0000 7fb9f9a4b700 1 -- 172.21.15.104:0/2737248093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f000b200 con 0x7fb9f40703f0 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.649+0000 7fb9f9a4b700 1 --2- 172.21.15.104:0/2737248093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f410cb80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb9f0011ed0 tx=0x7fb9f0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.649+0000 7fb9e27fc700 1 -- 172.21.15.104:0/2737248093 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9f001b570 con 0x7fb9f40703f0 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.649+0000 7fb9faa4d700 1 -- 172.21.15.104:0/2737248093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f410d270 con 0x7fb9f40703f0 2024-02-12T16:29:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.649+0000 7fb9e27fc700 1 -- 172.21.15.104:0/2737248093 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb9f001bb60 con 0x7fb9f40703f0 2024-02-12T16:29:11.654 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.649+0000 7fb9e27fc700 1 -- 172.21.15.104:0/2737248093 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9f001aad0 con 0x7fb9f40703f0 2024-02-12T16:29:11.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.649+0000 7fb9faa4d700 1 -- 172.21.15.104:0/2737248093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f410d700 con 0x7fb9f40703f0 2024-02-12T16:29:11.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.650+0000 7fb9e27fc700 1 -- 172.21.15.104:0/2737248093 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb9f000f070 con 0x7fb9f40703f0 2024-02-12T16:29:11.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.650+0000 7fb9faa4d700 1 -- 172.21.15.104:0/2737248093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d4004fa0 con 0x7fb9f40703f0 2024-02-12T16:29:11.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.650+0000 7fb9e27fc700 1 --2- 172.21.15.104:0/2737248093 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fb9dc02ff20 0x7fb9dc0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:11.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.650+0000 7fb9e27fc700 1 -- 172.21.15.104:0/2737248093 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb9f00414a0 con 0x7fb9f40703f0 2024-02-12T16:29:11.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.650+0000 7fb9f924a700 1 --2- 172.21.15.104:0/2737248093 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fb9dc02ff20 0x7fb9dc0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:11.655 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.650+0000 7fb9f924a700 1 --2- 172.21.15.104:0/2737248093 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fb9dc02ff20 0x7fb9dc0323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb9e40096a0 tx=0x7fb9e4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:11.656 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.654+0000 7fb9e27fc700 1 -- 172.21.15.104:0/2737248093 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9f0022ac0 con 0x7fb9f40703f0 2024-02-12T16:29:11.832 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.829+0000 7fb9faa4d700 1 -- 172.21.15.104:0/2737248093 --> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9d4000bc0 con 0x7fb9dc02ff20 2024-02-12T16:29:11.835 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.833+0000 7fb9e27fc700 1 -- 172.21.15.104:0/2737248093 <== mgr.14142 v2:172.21.15.104:6800/3327302611 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fb9d4000bc0 con 0x7fb9dc02ff20 2024-02-12T16:29:11.835 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-12T16:29:11.838 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.835+0000 7fb9faa4d700 1 -- 172.21.15.104:0/2737248093 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fb9dc02ff20 msgr2=0x7fb9dc0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:11.838 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.835+0000 7fb9faa4d700 1 --2- 172.21.15.104:0/2737248093 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fb9dc02ff20 0x7fb9dc0323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb9e40096a0 tx=0x7fb9e4006b40).stop 2024-02-12T16:29:11.838 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.835+0000 7fb9faa4d700 1 -- 172.21.15.104:0/2737248093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 msgr2=0x7fb9f410cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:11.838 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.835+0000 7fb9faa4d700 1 --2- 172.21.15.104:0/2737248093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f410cb80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb9f0011ed0 tx=0x7fb9f0006ee0).stop 2024-02-12T16:29:11.838 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.835+0000 7fb9faa4d700 1 -- 172.21.15.104:0/2737248093 shutdown_connections 2024-02-12T16:29:11.838 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.835+0000 7fb9faa4d700 1 --2- 172.21.15.104:0/2737248093 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fb9dc02ff20 0x7fb9dc0323e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:11.838 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.835+0000 7fb9faa4d700 1 --2- 172.21.15.104:0/2737248093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f40703f0 0x7fb9f410cb80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:11.838 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.835+0000 7fb9faa4d700 1 -- 172.21.15.104:0/2737248093 >> 172.21.15.104:0/2737248093 conn(0x7fb9f40f6390 msgr2=0x7fb9f40f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:11.838 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.836+0000 7fb9faa4d700 1 -- 172.21.15.104:0/2737248093 shutdown_connections 2024-02-12T16:29:11.838 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:11.836+0000 7fb9faa4d700 1 -- 172.21.15.104:0/2737248093 wait complete. 2024-02-12T16:29:12.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:10.421782+0000 mgr.smithi104.ohwbhf (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:29:12.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:29:10.422314+0000 mgr.smithi104.ohwbhf (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-12T16:29:12.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:11.834423+0000 mon.smithi104 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:12.443 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying alertmanager service with default placement... 2024-02-12T16:29:12.443 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-12T16:29:13.033 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.027+0000 7feefa6f7700 1 Processor -- start 2024-02-12T16:29:13.033 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.027+0000 7feefa6f7700 1 -- start start 2024-02-12T16:29:13.033 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.027+0000 7feefa6f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef40fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:13.033 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.027+0000 7feefa6f7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef40fc7b0 con 0x7feef40fbea0 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.027+0000 7feef3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef40fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.027+0000 7feef3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef40fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43184/0 (socket says 172.21.15.104:43184) 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.027+0000 7feef3fff700 1 -- 172.21.15.104:0/426625892 learned_addr learned my addr 172.21.15.104:0/426625892 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.027+0000 7feef3fff700 1 -- 172.21.15.104:0/426625892 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef40fc8f0 con 0x7feef40fbea0 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.027+0000 7feef3fff700 1 --2- 172.21.15.104:0/426625892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef40fc2c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7feee400b5f0 tx=0x7feee4006ee0).ready entity=mon.0 client_cookie=90ed3a37c95fdc64 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.028+0000 7feef2ffd700 1 -- 172.21.15.104:0/426625892 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee400d600 con 0x7feef40fbea0 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.028+0000 7feef2ffd700 1 -- 172.21.15.104:0/426625892 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7feee400dbf0 con 0x7feef40fbea0 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.028+0000 7feef2ffd700 1 -- 172.21.15.104:0/426625892 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee4011cb0 con 0x7feef40fbea0 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.028+0000 7feefa6f7700 1 -- 172.21.15.104:0/426625892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 msgr2=0x7feef40fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.028+0000 7feefa6f7700 1 --2- 172.21.15.104:0/426625892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef40fc2c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7feee400b5f0 tx=0x7feee4006ee0).stop 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.028+0000 7feefa6f7700 1 -- 172.21.15.104:0/426625892 shutdown_connections 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.028+0000 7feefa6f7700 1 --2- 172.21.15.104:0/426625892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef40fc2c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.028+0000 7feefa6f7700 1 -- 172.21.15.104:0/426625892 >> 172.21.15.104:0/426625892 conn(0x7feef40f76a0 msgr2=0x7feef40f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:13.034 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.028+0000 7feefa6f7700 1 -- 172.21.15.104:0/426625892 shutdown_connections 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.028+0000 7feefa6f7700 1 -- 172.21.15.104:0/426625892 wait complete. 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.029+0000 7feefa6f7700 1 Processor -- start 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.029+0000 7feefa6f7700 1 -- start start 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.029+0000 7feefa6f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.029+0000 7feefa6f7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef410b080 con 0x7feef40fbea0 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.029+0000 7feef3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.029+0000 7feef3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef410ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43198/0 (socket says 172.21.15.104:43198) 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.029+0000 7feef3fff700 1 -- 172.21.15.104:0/2201922851 learned_addr learned my addr 172.21.15.104:0/2201922851 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.029+0000 7feef3fff700 1 -- 172.21.15.104:0/2201922851 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee400b200 con 0x7feef40fbea0 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.030+0000 7feef3fff700 1 --2- 172.21.15.104:0/2201922851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef410ab90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7feee400bf40 tx=0x7feee4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.030+0000 7feef0ff9700 1 -- 172.21.15.104:0/2201922851 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee401b570 con 0x7feef40fbea0 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.030+0000 7feefa6f7700 1 -- 172.21.15.104:0/2201922851 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef410b280 con 0x7feef40fbea0 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.030+0000 7feef0ff9700 1 -- 172.21.15.104:0/2201922851 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7feee401bb60 con 0x7feef40fbea0 2024-02-12T16:29:13.035 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.030+0000 7feef0ff9700 1 -- 172.21.15.104:0/2201922851 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee401aa20 con 0x7feef40fbea0 2024-02-12T16:29:13.036 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.030+0000 7feefa6f7700 1 -- 172.21.15.104:0/2201922851 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef410b710 con 0x7feef40fbea0 2024-02-12T16:29:13.036 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.031+0000 7feefa6f7700 1 -- 172.21.15.104:0/2201922851 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef4045ad0 con 0x7feef40fbea0 2024-02-12T16:29:13.036 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.031+0000 7feef0ff9700 1 -- 172.21.15.104:0/2201922851 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7feee400f070 con 0x7feef40fbea0 2024-02-12T16:29:13.036 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.031+0000 7feef0ff9700 1 --2- 172.21.15.104:0/2201922851 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7feee002ff20 0x7feee00323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:13.036 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.031+0000 7feef0ff9700 1 -- 172.21.15.104:0/2201922851 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7feee4041680 con 0x7feef40fbea0 2024-02-12T16:29:13.036 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.031+0000 7feef37fe700 1 --2- 172.21.15.104:0/2201922851 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7feee002ff20 0x7feee00323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:13.036 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.031+0000 7feef37fe700 1 --2- 172.21.15.104:0/2201922851 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7feee002ff20 0x7feee00323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7feee80096a0 tx=0x7feee8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:13.037 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.035+0000 7feef0ff9700 1 -- 172.21.15.104:0/2201922851 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feee4019cd0 con 0x7feef40fbea0 2024-02-12T16:29:13.213 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.211+0000 7feefa6f7700 1 -- 172.21.15.104:0/2201922851 --> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7feef410c030 con 0x7feee002ff20 2024-02-12T16:29:13.216 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.214+0000 7feef0ff9700 1 -- 172.21.15.104:0/2201922851 <== mgr.14142 v2:172.21.15.104:6800/3327302611 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7feef410c030 con 0x7feee002ff20 2024-02-12T16:29:13.216 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-12T16:29:13.219 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.216+0000 7feefa6f7700 1 -- 172.21.15.104:0/2201922851 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7feee002ff20 msgr2=0x7feee00323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:13.219 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.216+0000 7feefa6f7700 1 --2- 172.21.15.104:0/2201922851 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7feee002ff20 0x7feee00323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7feee80096a0 tx=0x7feee8006b40).stop 2024-02-12T16:29:13.219 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.216+0000 7feefa6f7700 1 -- 172.21.15.104:0/2201922851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 msgr2=0x7feef410ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:13.219 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.216+0000 7feefa6f7700 1 --2- 172.21.15.104:0/2201922851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef410ab90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7feee400bf40 tx=0x7feee4006ee0).stop 2024-02-12T16:29:13.219 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.216+0000 7feefa6f7700 1 -- 172.21.15.104:0/2201922851 shutdown_connections 2024-02-12T16:29:13.219 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.216+0000 7feefa6f7700 1 --2- 172.21.15.104:0/2201922851 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7feee002ff20 0x7feee00323e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:13.219 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.216+0000 7feefa6f7700 1 --2- 172.21.15.104:0/2201922851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feef40fbea0 0x7feef410ab90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:13.220 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.216+0000 7feefa6f7700 1 -- 172.21.15.104:0/2201922851 >> 172.21.15.104:0/2201922851 conn(0x7feef40f76a0 msgr2=0x7feef40f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:13.220 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.216+0000 7feefa6f7700 1 -- 172.21.15.104:0/2201922851 shutdown_connections 2024-02-12T16:29:13.220 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:13.216+0000 7feefa6f7700 1 -- 172.21.15.104:0/2201922851 wait complete. 2024-02-12T16:29:13.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:11.831576+0000 mgr.smithi104.ohwbhf (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:29:13.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:29:11.832242+0000 mgr.smithi104.ohwbhf (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-12T16:29:13.924 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-12T16:29:14.505 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.498+0000 7f1746315700 1 Processor -- start 2024-02-12T16:29:14.505 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.498+0000 7f1746315700 1 -- start start 2024-02-12T16:29:14.505 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.498+0000 7f1746315700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f17400fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:14.505 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.498+0000 7f1746315700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17400fb6a0 con 0x7f17400fad90 2024-02-12T16:29:14.505 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.499+0000 7f173ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f17400fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:14.505 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.499+0000 7f173ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f17400fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43208/0 (socket says 172.21.15.104:43208) 2024-02-12T16:29:14.505 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.499+0000 7f173ffff700 1 -- 172.21.15.104:0/2410373401 learned_addr learned my addr 172.21.15.104:0/2410373401 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:14.505 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.499+0000 7f173ffff700 1 -- 172.21.15.104:0/2410373401 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17400fb7e0 con 0x7f17400fad90 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.499+0000 7f173ffff700 1 --2- 172.21.15.104:0/2410373401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f17400fb1b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f173400b5f0 tx=0x7f1734006ee0).ready entity=mon.0 client_cookie=adf9d9c924ad8ebf server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.499+0000 7f173effd700 1 -- 172.21.15.104:0/2410373401 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f173400d600 con 0x7f17400fad90 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.499+0000 7f173effd700 1 -- 172.21.15.104:0/2410373401 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f173400dbf0 con 0x7f17400fad90 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.499+0000 7f173effd700 1 -- 172.21.15.104:0/2410373401 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1734011cb0 con 0x7f17400fad90 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.500+0000 7f1746315700 1 -- 172.21.15.104:0/2410373401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 msgr2=0x7f17400fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.500+0000 7f1746315700 1 --2- 172.21.15.104:0/2410373401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f17400fb1b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f173400b5f0 tx=0x7f1734006ee0).stop 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.500+0000 7f1746315700 1 -- 172.21.15.104:0/2410373401 shutdown_connections 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.500+0000 7f1746315700 1 --2- 172.21.15.104:0/2410373401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f17400fb1b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.500+0000 7f1746315700 1 -- 172.21.15.104:0/2410373401 >> 172.21.15.104:0/2410373401 conn(0x7f17400f6530 msgr2=0x7f17400f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.500+0000 7f1746315700 1 -- 172.21.15.104:0/2410373401 shutdown_connections 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.500+0000 7f1746315700 1 -- 172.21.15.104:0/2410373401 wait complete. 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.500+0000 7f1746315700 1 Processor -- start 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f1746315700 1 -- start start 2024-02-12T16:29:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f1746315700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f174010d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f1746315700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1740070450 con 0x7f17400fad90 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f173ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f174010d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f173ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f174010d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43216/0 (socket says 172.21.15.104:43216) 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f173ffff700 1 -- 172.21.15.104:0/2112492068 learned_addr learned my addr 172.21.15.104:0/2112492068 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f173ffff700 1 -- 172.21.15.104:0/2112492068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173400b200 con 0x7f17400fad90 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f173ffff700 1 --2- 172.21.15.104:0/2112492068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f174010d860 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1734011ea0 tx=0x7f1734006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f173cff9700 1 -- 172.21.15.104:0/2112492068 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f173401b570 con 0x7f17400fad90 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f1746315700 1 -- 172.21.15.104:0/2112492068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1740070650 con 0x7f17400fad90 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f173cff9700 1 -- 172.21.15.104:0/2112492068 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f173401bb60 con 0x7f17400fad90 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f173cff9700 1 -- 172.21.15.104:0/2112492068 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f173401a8d0 con 0x7f17400fad90 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.501+0000 7f1746315700 1 -- 172.21.15.104:0/2112492068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174010df10 con 0x7f17400fad90 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.502+0000 7f173cff9700 1 -- 172.21.15.104:0/2112492068 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f173400f070 con 0x7f17400fad90 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.502+0000 7f1746315700 1 -- 172.21.15.104:0/2112492068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1740045ad0 con 0x7f17400fad90 2024-02-12T16:29:14.507 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.502+0000 7f173cff9700 1 --2- 172.21.15.104:0/2112492068 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f172c02ff20 0x7f172c0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:14.508 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.502+0000 7f173cff9700 1 -- 172.21.15.104:0/2112492068 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1734041680 con 0x7f17400fad90 2024-02-12T16:29:14.508 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.503+0000 7f173f7fe700 1 --2- 172.21.15.104:0/2112492068 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f172c02ff20 0x7f172c0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:14.508 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.503+0000 7f173f7fe700 1 --2- 172.21.15.104:0/2112492068 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f172c02ff20 0x7f172c0323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f17300096a0 tx=0x7f1730006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:14.508 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.506+0000 7f173cff9700 1 -- 172.21.15.104:0/2112492068 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1734022d20 con 0x7f17400fad90 2024-02-12T16:29:14.657 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.655+0000 7f1746315700 1 -- 172.21.15.104:0/2112492068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f1740027a00 con 0x7f17400fad90 2024-02-12T16:29:14.663 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.660+0000 7f173cff9700 1 -- 172.21.15.104:0/2112492068 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f1734016070 con 0x7f17400fad90 2024-02-12T16:29:14.665 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.662+0000 7f1746315700 1 -- 172.21.15.104:0/2112492068 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f172c02ff20 msgr2=0x7f172c0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:14.666 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.662+0000 7f1746315700 1 --2- 172.21.15.104:0/2112492068 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f172c02ff20 0x7f172c0323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f17300096a0 tx=0x7f1730006b40).stop 2024-02-12T16:29:14.666 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.662+0000 7f1746315700 1 -- 172.21.15.104:0/2112492068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 msgr2=0x7f174010d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:14.666 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.662+0000 7f1746315700 1 --2- 172.21.15.104:0/2112492068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f174010d860 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1734011ea0 tx=0x7f1734006ee0).stop 2024-02-12T16:29:14.666 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.663+0000 7f1746315700 1 -- 172.21.15.104:0/2112492068 shutdown_connections 2024-02-12T16:29:14.666 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.663+0000 7f1746315700 1 --2- 172.21.15.104:0/2112492068 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f172c02ff20 0x7f172c0323e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:14.666 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.663+0000 7f1746315700 1 --2- 172.21.15.104:0/2112492068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f17400fad90 0x7f174010d860 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:14.666 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.663+0000 7f1746315700 1 -- 172.21.15.104:0/2112492068 >> 172.21.15.104:0/2112492068 conn(0x7f17400f6530 msgr2=0x7f17400f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:14.666 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.663+0000 7f1746315700 1 -- 172.21.15.104:0/2112492068 shutdown_connections 2024-02-12T16:29:14.666 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:14.663+0000 7f1746315700 1 -- 172.21.15.104:0/2112492068 wait complete. 2024-02-12T16:29:14.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:13.212761+0000 mgr.smithi104.ohwbhf (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:29:14.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:29:13.213287+0000 mgr.smithi104.ohwbhf (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-12T16:29:14.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:13.215465+0000 mon.smithi104 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.104:0/1040382084' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:15.311 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-12T16:29:15.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:15 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:14.661968+0000 mon.smithi104 (mon.0) 82 : audit [INF] from='client.? 172.21.15.104:0/2112492068' entity='client.admin' 2024-02-12T16:29:17.053 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.045+0000 7fce724c5700 1 Processor -- start 2024-02-12T16:29:17.053 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.045+0000 7fce724c5700 1 -- start start 2024-02-12T16:29:17.053 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.045+0000 7fce724c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:17.053 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.045+0000 7fce724c5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0fb6a0 con 0x7fce6c0fad90 2024-02-12T16:29:17.053 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.045+0000 7fce6bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:17.053 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.046+0000 7fce6bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43228/0 (socket says 172.21.15.104:43228) 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.046+0000 7fce6bfff700 1 -- 172.21.15.104:0/1267080069 learned_addr learned my addr 172.21.15.104:0/1267080069 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.046+0000 7fce6bfff700 1 -- 172.21.15.104:0/1267080069 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c0fb7e0 con 0x7fce6c0fad90 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.046+0000 7fce6bfff700 1 --2- 172.21.15.104:0/1267080069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c0fb1b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fce6000b5f0 tx=0x7fce60006ee0).ready entity=mon.0 client_cookie=28b2ee8bd3cb03de server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.046+0000 7fce6affd700 1 -- 172.21.15.104:0/1267080069 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce6000d600 con 0x7fce6c0fad90 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.046+0000 7fce6affd700 1 -- 172.21.15.104:0/1267080069 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fce6000dbf0 con 0x7fce6c0fad90 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.046+0000 7fce6affd700 1 -- 172.21.15.104:0/1267080069 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce60011cb0 con 0x7fce6c0fad90 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.047+0000 7fce724c5700 1 -- 172.21.15.104:0/1267080069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 msgr2=0x7fce6c0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.047+0000 7fce724c5700 1 --2- 172.21.15.104:0/1267080069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c0fb1b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fce6000b5f0 tx=0x7fce60006ee0).stop 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.047+0000 7fce724c5700 1 -- 172.21.15.104:0/1267080069 shutdown_connections 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.047+0000 7fce724c5700 1 --2- 172.21.15.104:0/1267080069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c0fb1b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.047+0000 7fce724c5700 1 -- 172.21.15.104:0/1267080069 >> 172.21.15.104:0/1267080069 conn(0x7fce6c0f6530 msgr2=0x7fce6c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.047+0000 7fce724c5700 1 -- 172.21.15.104:0/1267080069 shutdown_connections 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.047+0000 7fce724c5700 1 -- 172.21.15.104:0/1267080069 wait complete. 2024-02-12T16:29:17.054 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.047+0000 7fce724c5700 1 Processor -- start 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.047+0000 7fce724c5700 1 -- start start 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce724c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c10cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce724c5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c10d270 con 0x7fce6c0fad90 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce6bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c10cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce6bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c10cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43232/0 (socket says 172.21.15.104:43232) 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce6bfff700 1 -- 172.21.15.104:0/965444448 learned_addr learned my addr 172.21.15.104:0/965444448 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce6bfff700 1 -- 172.21.15.104:0/965444448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6000b200 con 0x7fce6c0fad90 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce6bfff700 1 --2- 172.21.15.104:0/965444448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c10cd80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fce60011ea0 tx=0x7fce60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce68ff9700 1 -- 172.21.15.104:0/965444448 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce6001b570 con 0x7fce6c0fad90 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce724c5700 1 -- 172.21.15.104:0/965444448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6c10d470 con 0x7fce6c0fad90 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce68ff9700 1 -- 172.21.15.104:0/965444448 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fce6001bb60 con 0x7fce6c0fad90 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce68ff9700 1 -- 172.21.15.104:0/965444448 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce6001a8d0 con 0x7fce6c0fad90 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.048+0000 7fce724c5700 1 -- 172.21.15.104:0/965444448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6c10e080 con 0x7fce6c0fad90 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.049+0000 7fce68ff9700 1 -- 172.21.15.104:0/965444448 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fce6000f070 con 0x7fce6c0fad90 2024-02-12T16:29:17.055 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.049+0000 7fce724c5700 1 -- 172.21.15.104:0/965444448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6c059100 con 0x7fce6c0fad90 2024-02-12T16:29:17.056 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.049+0000 7fce68ff9700 1 --2- 172.21.15.104:0/965444448 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fce5802ff00 0x7fce580323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:17.056 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.049+0000 7fce68ff9700 1 -- 172.21.15.104:0/965444448 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fce600415b0 con 0x7fce6c0fad90 2024-02-12T16:29:17.056 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.050+0000 7fce6b7fe700 1 --2- 172.21.15.104:0/965444448 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fce5802ff00 0x7fce580323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:17.056 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.050+0000 7fce6b7fe700 1 --2- 172.21.15.104:0/965444448 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fce5802ff00 0x7fce580323c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fce5c0096a0 tx=0x7fce5c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:17.056 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.054+0000 7fce68ff9700 1 -- 172.21.15.104:0/965444448 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce60019c90 con 0x7fce6c0fad90 2024-02-12T16:29:17.225 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.223+0000 7fce724c5700 1 -- 172.21.15.104:0/965444448 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fce6c10eeb0 con 0x7fce6c0fad90 2024-02-12T16:29:17.231 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.229+0000 7fce68ff9700 1 -- 172.21.15.104:0/965444448 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fce60016070 con 0x7fce6c0fad90 2024-02-12T16:29:17.233 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.230+0000 7fce724c5700 1 -- 172.21.15.104:0/965444448 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fce5802ff00 msgr2=0x7fce580323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:17.233 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.230+0000 7fce724c5700 1 --2- 172.21.15.104:0/965444448 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fce5802ff00 0x7fce580323c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fce5c0096a0 tx=0x7fce5c006b40).stop 2024-02-12T16:29:17.233 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.230+0000 7fce724c5700 1 -- 172.21.15.104:0/965444448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 msgr2=0x7fce6c10cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:17.234 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.230+0000 7fce724c5700 1 --2- 172.21.15.104:0/965444448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c10cd80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fce60011ea0 tx=0x7fce60006ee0).stop 2024-02-12T16:29:17.234 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.230+0000 7fce724c5700 1 -- 172.21.15.104:0/965444448 shutdown_connections 2024-02-12T16:29:17.234 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.230+0000 7fce724c5700 1 --2- 172.21.15.104:0/965444448 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fce5802ff00 0x7fce580323c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:17.234 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.230+0000 7fce724c5700 1 --2- 172.21.15.104:0/965444448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce6c0fad90 0x7fce6c10cd80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:17.234 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.230+0000 7fce724c5700 1 -- 172.21.15.104:0/965444448 >> 172.21.15.104:0/965444448 conn(0x7fce6c0f6530 msgr2=0x7fce6c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:17.234 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.231+0000 7fce724c5700 1 -- 172.21.15.104:0/965444448 shutdown_connections 2024-02-12T16:29:17.234 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:17.231+0000 7fce724c5700 1 -- 172.21.15.104:0/965444448 wait complete. 2024-02-12T16:29:18.127 INFO:teuthology.orchestra.run.smithi104.stderr:Enabling the dashboard module... 2024-02-12T16:29:18.127 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-12T16:29:18.396 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:17.230150+0000 mon.smithi104 (mon.0) 83 : audit [INF] from='client.? 172.21.15.104:0/965444448' entity='client.admin' 2024-02-12T16:29:18.902 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.897+0000 7fe467b6e700 1 Processor -- start 2024-02-12T16:29:18.902 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.897+0000 7fe467b6e700 1 -- start start 2024-02-12T16:29:18.902 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.897+0000 7fe467b6e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe4600fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:18.902 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.897+0000 7fe467b6e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4600fef40 con 0x7fe460070430 2024-02-12T16:29:18.902 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.898+0000 7fe46590a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe4600fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:18.902 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.898+0000 7fe46590a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe4600fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43238/0 (socket says 172.21.15.104:43238) 2024-02-12T16:29:18.902 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.898+0000 7fe46590a700 1 -- 172.21.15.104:0/3026612397 learned_addr learned my addr 172.21.15.104:0/3026612397 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:18.902 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.898+0000 7fe46590a700 1 -- 172.21.15.104:0/3026612397 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4600ff080 con 0x7fe460070430 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.898+0000 7fe46590a700 1 --2- 172.21.15.104:0/3026612397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe4600fea50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe45000b5f0 tx=0x7fe450006ee0).ready entity=mon.0 client_cookie=c85949811a6fa049 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.898+0000 7fe464908700 1 -- 172.21.15.104:0/3026612397 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe45000d600 con 0x7fe460070430 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.898+0000 7fe464908700 1 -- 172.21.15.104:0/3026612397 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe45000dbf0 con 0x7fe460070430 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.898+0000 7fe464908700 1 -- 172.21.15.104:0/3026612397 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe450011cb0 con 0x7fe460070430 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.899+0000 7fe467b6e700 1 -- 172.21.15.104:0/3026612397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 msgr2=0x7fe4600fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.899+0000 7fe467b6e700 1 --2- 172.21.15.104:0/3026612397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe4600fea50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe45000b5f0 tx=0x7fe450006ee0).stop 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.899+0000 7fe467b6e700 1 -- 172.21.15.104:0/3026612397 shutdown_connections 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.899+0000 7fe467b6e700 1 --2- 172.21.15.104:0/3026612397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe4600fea50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.899+0000 7fe467b6e700 1 -- 172.21.15.104:0/3026612397 >> 172.21.15.104:0/3026612397 conn(0x7fe4600f6530 msgr2=0x7fe4600f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.899+0000 7fe467b6e700 1 -- 172.21.15.104:0/3026612397 shutdown_connections 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.899+0000 7fe467b6e700 1 -- 172.21.15.104:0/3026612397 wait complete. 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.899+0000 7fe467b6e700 1 Processor -- start 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.899+0000 7fe467b6e700 1 -- start start 2024-02-12T16:29:18.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe467b6e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe46010ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:18.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe467b6e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46010b020 con 0x7fe460070430 2024-02-12T16:29:18.905 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe46590a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe46010ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:18.905 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe46590a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe46010ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43250/0 (socket says 172.21.15.104:43250) 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe46590a700 1 -- 172.21.15.104:0/2946874843 learned_addr learned my addr 172.21.15.104:0/2946874843 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe46590a700 1 -- 172.21.15.104:0/2946874843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45000b200 con 0x7fe460070430 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe46590a700 1 --2- 172.21.15.104:0/2946874843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe46010ab30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe450006800 tx=0x7fe450006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe44e7fc700 1 -- 172.21.15.104:0/2946874843 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe45001b570 con 0x7fe460070430 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe467b6e700 1 -- 172.21.15.104:0/2946874843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe46010b220 con 0x7fe460070430 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe44e7fc700 1 -- 172.21.15.104:0/2946874843 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe45001bb60 con 0x7fe460070430 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe44e7fc700 1 -- 172.21.15.104:0/2946874843 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe45001a9f0 con 0x7fe460070430 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.900+0000 7fe467b6e700 1 -- 172.21.15.104:0/2946874843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46010b6b0 con 0x7fe460070430 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.901+0000 7fe44e7fc700 1 -- 172.21.15.104:0/2946874843 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fe45000f070 con 0x7fe460070430 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.901+0000 7fe44e7fc700 1 --2- 172.21.15.104:0/2946874843 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fe44802ff20 0x7fe4480323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.901+0000 7fe44e7fc700 1 -- 172.21.15.104:0/2946874843 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe4500415d0 con 0x7fe460070430 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.901+0000 7fe467b6e700 1 -- 172.21.15.104:0/2946874843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe460045ad0 con 0x7fe460070430 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.901+0000 7fe465109700 1 --2- 172.21.15.104:0/2946874843 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fe44802ff20 0x7fe4480323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:18.906 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.902+0000 7fe465109700 1 --2- 172.21.15.104:0/2946874843 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fe44802ff20 0x7fe4480323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe45c0096a0 tx=0x7fe45c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:18.907 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:18.905+0000 7fe44e7fc700 1 -- 172.21.15.104:0/2946874843 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe450010360 con 0x7fe460070430 2024-02-12T16:29:19.099 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.097+0000 7fe467b6e700 1 -- 172.21.15.104:0/2946874843 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fe460104c70 con 0x7fe460070430 2024-02-12T16:29:19.234 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.231+0000 7fe44e7fc700 1 -- 172.21.15.104:0/2946874843 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fe45001bcd0 con 0x7fe460070430 2024-02-12T16:29:19.234 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.232+0000 7fe44e7fc700 1 -- 172.21.15.104:0/2946874843 <== mon.0 v2:172.21.15.104:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7fe450005700 con 0x7fe460070430 2024-02-12T16:29:19.237 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.234+0000 7fe467b6e700 1 -- 172.21.15.104:0/2946874843 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fe44802ff20 msgr2=0x7fe4480323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:19.237 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.234+0000 7fe467b6e700 1 --2- 172.21.15.104:0/2946874843 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fe44802ff20 0x7fe4480323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe45c0096a0 tx=0x7fe45c006b40).stop 2024-02-12T16:29:19.237 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.234+0000 7fe467b6e700 1 -- 172.21.15.104:0/2946874843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 msgr2=0x7fe46010ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:19.237 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.234+0000 7fe467b6e700 1 --2- 172.21.15.104:0/2946874843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe46010ab30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe450006800 tx=0x7fe450006ee0).stop 2024-02-12T16:29:19.237 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.234+0000 7fe467b6e700 1 -- 172.21.15.104:0/2946874843 shutdown_connections 2024-02-12T16:29:19.237 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.234+0000 7fe467b6e700 1 --2- 172.21.15.104:0/2946874843 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7fe44802ff20 0x7fe4480323e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:19.237 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.234+0000 7fe467b6e700 1 --2- 172.21.15.104:0/2946874843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe460070430 0x7fe46010ab30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:19.237 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.234+0000 7fe467b6e700 1 -- 172.21.15.104:0/2946874843 >> 172.21.15.104:0/2946874843 conn(0x7fe4600f6530 msgr2=0x7fe4600f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:19.237 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.234+0000 7fe467b6e700 1 -- 172.21.15.104:0/2946874843 shutdown_connections 2024-02-12T16:29:19.238 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:19.234+0000 7fe467b6e700 1 -- 172.21.15.104:0/2946874843 wait complete. 2024-02-12T16:29:19.378 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:19 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:19.098677+0000 mon.smithi104 (mon.0) 84 : audit [INF] from='client.? 172.21.15.104:0/2946874843' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-12T16:29:19.955 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-12T16:29:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.604+0000 7f21b9338700 1 Processor -- start 2024-02-12T16:29:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.604+0000 7f21b9338700 1 -- start start 2024-02-12T16:29:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.604+0000 7f21b9338700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b40faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.604+0000 7f21b9338700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40fb440 con 0x7f21b40fab30 2024-02-12T16:29:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.604+0000 7f21b3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b40faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.604+0000 7f21b3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b40faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43284/0 (socket says 172.21.15.104:43284) 2024-02-12T16:29:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.604+0000 7f21b3fff700 1 -- 172.21.15.104:0/721093950 learned_addr learned my addr 172.21.15.104:0/721093950 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.605+0000 7f21b3fff700 1 -- 172.21.15.104:0/721093950 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b40fb580 con 0x7f21b40fab30 2024-02-12T16:29:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.605+0000 7f21b3fff700 1 --2- 172.21.15.104:0/721093950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b40faf50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f21a400b5f0 tx=0x7f21a4006ee0).ready entity=mon.0 client_cookie=f538701f43b45547 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.605+0000 7f21b2ffd700 1 -- 172.21.15.104:0/721093950 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21a400d600 con 0x7f21b40fab30 2024-02-12T16:29:20.608 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.605+0000 7f21b2ffd700 1 -- 172.21.15.104:0/721093950 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f21a400dbf0 con 0x7f21b40fab30 2024-02-12T16:29:20.609 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.605+0000 7f21b2ffd700 1 -- 172.21.15.104:0/721093950 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21a4011cb0 con 0x7f21b40fab30 2024-02-12T16:29:20.609 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.606+0000 7f21b9338700 1 -- 172.21.15.104:0/721093950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 msgr2=0x7f21b40faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:20.609 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.606+0000 7f21b9338700 1 --2- 172.21.15.104:0/721093950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b40faf50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f21a400b5f0 tx=0x7f21a4006ee0).stop 2024-02-12T16:29:20.611 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.606+0000 7f21b9338700 1 -- 172.21.15.104:0/721093950 shutdown_connections 2024-02-12T16:29:20.611 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.606+0000 7f21b9338700 1 --2- 172.21.15.104:0/721093950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b40faf50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:20.611 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.606+0000 7f21b9338700 1 -- 172.21.15.104:0/721093950 >> 172.21.15.104:0/721093950 conn(0x7f21b40f6390 msgr2=0x7f21b40f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:20.611 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.606+0000 7f21b9338700 1 -- 172.21.15.104:0/721093950 shutdown_connections 2024-02-12T16:29:20.611 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.606+0000 7f21b9338700 1 -- 172.21.15.104:0/721093950 wait complete. 2024-02-12T16:29:20.611 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.606+0000 7f21b9338700 1 Processor -- start 2024-02-12T16:29:20.611 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.607+0000 7f21b9338700 1 -- start start 2024-02-12T16:29:20.611 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.607+0000 7f21b9338700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b410cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:20.611 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.607+0000 7f21b9338700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b410d050 con 0x7f21b40fab30 2024-02-12T16:29:20.611 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.607+0000 7f21b3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b410cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:20.611 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.607+0000 7f21b3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b410cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43296/0 (socket says 172.21.15.104:43296) 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.607+0000 7f21b3fff700 1 -- 172.21.15.104:0/264917114 learned_addr learned my addr 172.21.15.104:0/264917114 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.607+0000 7f21b3fff700 1 -- 172.21.15.104:0/264917114 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a400b200 con 0x7f21b40fab30 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.607+0000 7f21b3fff700 1 --2- 172.21.15.104:0/264917114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b410cb60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f21a4011ea0 tx=0x7f21a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.608+0000 7f21b0ff9700 1 -- 172.21.15.104:0/264917114 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21a401b570 con 0x7f21b40fab30 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.608+0000 7f21b9338700 1 -- 172.21.15.104:0/264917114 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b410d250 con 0x7f21b40fab30 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.608+0000 7f21b0ff9700 1 -- 172.21.15.104:0/264917114 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f21a401bb60 con 0x7f21b40fab30 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.608+0000 7f21b0ff9700 1 -- 172.21.15.104:0/264917114 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21a401a8d0 con 0x7f21b40fab30 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.608+0000 7f21b9338700 1 -- 172.21.15.104:0/264917114 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b410d6e0 con 0x7f21b40fab30 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.608+0000 7f21b0ff9700 1 -- 172.21.15.104:0/264917114 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f21a400f070 con 0x7f21b40fab30 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.608+0000 7f21b0ff9700 1 --2- 172.21.15.104:0/264917114 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f21a002ff50 0x7f21a0032410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.609+0000 7f21b0ff9700 1 -- 172.21.15.104:0/264917114 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f21a4042550 con 0x7f21b40fab30 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.609+0000 7f21b37fe700 1 -- 172.21.15.104:0/264917114 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f21a002ff50 msgr2=0x7f21a0032410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/3327302611 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.609+0000 7f21b37fe700 1 --2- 172.21.15.104:0/264917114 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f21a002ff50 0x7f21a0032410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-12T16:29:20.612 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.609+0000 7f21b9338700 1 -- 172.21.15.104:0/264917114 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b4045ad0 con 0x7f21b40fab30 2024-02-12T16:29:20.615 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.613+0000 7f21b0ff9700 1 -- 172.21.15.104:0/264917114 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21a4016070 con 0x7f21b40fab30 2024-02-12T16:29:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:19.233515+0000 mon.smithi104 (mon.0) 85 : audit [INF] from='client.? 172.21.15.104:0/2946874843' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-12T16:29:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:19.233563+0000 mon.smithi104 (mon.0) 86 : cluster [DBG] mgrmap e13: smithi104.ohwbhf(active, since 11s) 2024-02-12T16:29:20.810 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.807+0000 7f21b9338700 1 -- 172.21.15.104:0/264917114 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f21b410ec20 con 0x7f21b40fab30 2024-02-12T16:29:20.810 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.807+0000 7f21b0ff9700 1 -- 172.21.15.104:0/264917114 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f21a4022780 con 0x7f21b40fab30 2024-02-12T16:29:20.810 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-02-12T16:29:20.810 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-12T16:29:20.810 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "available": true, 2024-02-12T16:29:20.810 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "active_name": "smithi104.ohwbhf", 2024-02-12T16:29:20.810 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-12T16:29:20.810 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-02-12T16:29:20.811 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.809+0000 7f21b9338700 1 -- 172.21.15.104:0/264917114 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f21a002ff50 msgr2=0x7f21a0032410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-12T16:29:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.809+0000 7f21b9338700 1 --2- 172.21.15.104:0/264917114 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f21a002ff50 0x7f21a0032410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-12T16:29:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.809+0000 7f21b9338700 1 -- 172.21.15.104:0/264917114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 msgr2=0x7f21b410cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.809+0000 7f21b9338700 1 --2- 172.21.15.104:0/264917114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b410cb60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f21a4011ea0 tx=0x7f21a4006ee0).stop 2024-02-12T16:29:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.809+0000 7f21b9338700 1 -- 172.21.15.104:0/264917114 shutdown_connections 2024-02-12T16:29:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.809+0000 7f21b9338700 1 --2- 172.21.15.104:0/264917114 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7f21a002ff50 0x7f21a0032410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-12T16:29:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.809+0000 7f21b9338700 1 --2- 172.21.15.104:0/264917114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b40fab30 0x7f21b410cb60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.809+0000 7f21b9338700 1 -- 172.21.15.104:0/264917114 >> 172.21.15.104:0/264917114 conn(0x7f21b40f6390 msgr2=0x7f21b40ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.809+0000 7f21b9338700 1 -- 172.21.15.104:0/264917114 shutdown_connections 2024-02-12T16:29:20.812 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:20.809+0000 7f21b9338700 1 -- 172.21.15.104:0/264917114 wait complete. 2024-02-12T16:29:21.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:21 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:20.808373+0000 mon.smithi104 (mon.0) 87 : audit [DBG] from='client.? 172.21.15.104:0/264917114' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-12T16:29:21.510 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for the mgr to restart... 2024-02-12T16:29:21.511 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mgr epoch 13... 2024-02-12T16:29:21.511 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-12T16:29:22.221 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.214+0000 7ff35e6fe700 1 Processor -- start 2024-02-12T16:29:22.221 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.214+0000 7ff35e6fe700 1 -- start start 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.214+0000 7ff35e6fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff358100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.214+0000 7ff35e6fe700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3580fc8a0 con 0x7ff3580703d0 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.214+0000 7ff357fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff358100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.214+0000 7ff357fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff358100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43304/0 (socket says 172.21.15.104:43304) 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.214+0000 7ff357fff700 1 -- 172.21.15.104:0/3204488993 learned_addr learned my addr 172.21.15.104:0/3204488993 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.214+0000 7ff357fff700 1 -- 172.21.15.104:0/3204488993 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3581013a0 con 0x7ff3580703d0 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.214+0000 7ff357fff700 1 --2- 172.21.15.104:0/3204488993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff358100eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff34800b5f0 tx=0x7ff348006ee0).ready entity=mon.0 client_cookie=471ce2bd5661912c server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.214+0000 7ff356ffd700 1 -- 172.21.15.104:0/3204488993 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff34800d600 con 0x7ff3580703d0 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.215+0000 7ff356ffd700 1 -- 172.21.15.104:0/3204488993 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff34800dbf0 con 0x7ff3580703d0 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.215+0000 7ff356ffd700 1 -- 172.21.15.104:0/3204488993 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff348011cb0 con 0x7ff3580703d0 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.215+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3204488993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 msgr2=0x7ff358100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:22.222 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.215+0000 7ff35e6fe700 1 --2- 172.21.15.104:0/3204488993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff358100eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff34800b5f0 tx=0x7ff348006ee0).stop 2024-02-12T16:29:22.225 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.215+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3204488993 shutdown_connections 2024-02-12T16:29:22.225 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.215+0000 7ff35e6fe700 1 --2- 172.21.15.104:0/3204488993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff358100eb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:22.225 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.215+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3204488993 >> 172.21.15.104:0/3204488993 conn(0x7ff3580f6530 msgr2=0x7ff3580f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:22.225 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.215+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3204488993 shutdown_connections 2024-02-12T16:29:22.225 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.215+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3204488993 wait complete. 2024-02-12T16:29:22.225 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.216+0000 7ff35e6fe700 1 Processor -- start 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.216+0000 7ff35e6fe700 1 -- start start 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.216+0000 7ff35e6fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff35810cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.216+0000 7ff35e6fe700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35810d2a0 con 0x7ff3580703d0 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.216+0000 7ff357fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff35810cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.216+0000 7ff357fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff35810cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:43306/0 (socket says 172.21.15.104:43306) 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.216+0000 7ff357fff700 1 -- 172.21.15.104:0/3157121711 learned_addr learned my addr 172.21.15.104:0/3157121711 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.216+0000 7ff357fff700 1 -- 172.21.15.104:0/3157121711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34800b200 con 0x7ff3580703d0 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.216+0000 7ff357fff700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff35810cdb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff348006a20 tx=0x7ff348006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.217+0000 7ff354ff9700 1 -- 172.21.15.104:0/3157121711 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff34801b570 con 0x7ff3580703d0 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.217+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3157121711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35810d4a0 con 0x7ff3580703d0 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.217+0000 7ff354ff9700 1 -- 172.21.15.104:0/3157121711 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff34801bb60 con 0x7ff3580703d0 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.217+0000 7ff354ff9700 1 -- 172.21.15.104:0/3157121711 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff34801aa20 con 0x7ff3580703d0 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.217+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3157121711 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35810e0e0 con 0x7ff3580703d0 2024-02-12T16:29:22.226 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.217+0000 7ff354ff9700 1 -- 172.21.15.104:0/3157121711 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ff34800f070 con 0x7ff3580703d0 2024-02-12T16:29:22.227 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.218+0000 7ff354ff9700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 0x7ff344032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:22.227 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.218+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3157121711 --> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff358059100 con 0x7ff34402ff70 2024-02-12T16:29:22.227 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.218+0000 7ff354ff9700 1 -- 172.21.15.104:0/3157121711 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff348041670 con 0x7ff3580703d0 2024-02-12T16:29:22.227 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.218+0000 7ff3577fe700 1 -- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 msgr2=0x7ff344032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/3327302611 2024-02-12T16:29:22.227 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.218+0000 7ff3577fe700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 0x7ff344032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-12T16:29:22.421 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.419+0000 7ff3577fe700 1 -- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 msgr2=0x7ff344032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/3327302611 2024-02-12T16:29:22.421 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.419+0000 7ff3577fe700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 0x7ff344032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-12T16:29:22.822 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.820+0000 7ff3577fe700 1 -- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 msgr2=0x7ff344032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/3327302611 2024-02-12T16:29:22.822 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:22.820+0000 7ff3577fe700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 0x7ff344032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-12T16:29:23.623 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:23.621+0000 7ff3577fe700 1 -- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 msgr2=0x7ff344032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.104:6800/3327302611 2024-02-12T16:29:23.623 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:23.621+0000 7ff3577fe700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 0x7ff344032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-12T16:29:24.271 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:24.268+0000 7ff354ff9700 1 -- 172.21.15.104:0/3157121711 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7ff348025980 con 0x7ff3580703d0 2024-02-12T16:29:24.271 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:24.268+0000 7ff354ff9700 1 -- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 msgr2=0x7ff344032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-12T16:29:24.271 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:24.268+0000 7ff354ff9700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 0x7ff344032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-12T16:29:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:24.217050+0000 mon.smithi104 (mon.0) 88 : cluster [INF] Active manager daemon smithi104.ohwbhf restarted 2024-02-12T16:29:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:24.217431+0000 mon.smithi104 (mon.0) 89 : cluster [INF] Activating manager daemon smithi104.ohwbhf 2024-02-12T16:29:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:24.221540+0000 mon.smithi104 (mon.0) 90 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T16:29:25.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.274+0000 7ff354ff9700 1 -- 172.21.15.104:0/3157121711 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7ff348025460 con 0x7ff3580703d0 2024-02-12T16:29:25.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.274+0000 7ff354ff9700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff344033a50 0x7ff344035e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:25.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.274+0000 7ff354ff9700 1 -- 172.21.15.104:0/3157121711 --> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff358059100 con 0x7ff344033a50 2024-02-12T16:29:25.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.275+0000 7ff3577fe700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff344033a50 0x7ff344035e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:25.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.275+0000 7ff3577fe700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff344033a50 0x7ff344035e40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7ff34c0036d0 tx=0x7ff34c006ca0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:25.278 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.276+0000 7ff354ff9700 1 -- 172.21.15.104:0/3157121711 <== mgr.14162 v2:172.21.15.104:6800/4169661527 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7ff358059100 con 0x7ff344033a50 2024-02-12T16:29:25.284 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3157121711 --> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff35810d930 con 0x7ff344033a50 2024-02-12T16:29:25.285 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff354ff9700 1 -- 172.21.15.104:0/3157121711 <== mgr.14162 v2:172.21.15.104:6800/4169661527 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7ff35810d930 con 0x7ff344033a50 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "initialized": true 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff344033a50 msgr2=0x7ff344035e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff344033a50 0x7ff344035e40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7ff34c0036d0 tx=0x7ff34c006ca0).stop 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 msgr2=0x7ff35810cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff35810cdb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff348006a20 tx=0x7ff348006ee0).stop 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3157121711 shutdown_connections 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff344033a50 0x7ff344035e40 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3580703d0 0x7ff35810cdb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 --2- 172.21.15.104:0/3157121711 >> [v2:172.21.15.104:6800/3327302611,v1:172.21.15.104:6801/3327302611] conn(0x7ff34402ff70 0x7ff344032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3157121711 >> 172.21.15.104:0/3157121711 conn(0x7ff3580f6530 msgr2=0x7ff3580f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:25.286 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3157121711 shutdown_connections 2024-02-12T16:29:25.287 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:25.282+0000 7ff35e6fe700 1 -- 172.21.15.104:0/3157121711 wait complete. 2024-02-12T16:29:25.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:24.270219+0000 mon.smithi104 (mon.0) 91 : cluster [DBG] mgrmap e14: smithi104.ohwbhf(active, starting, since 0.0528701s) 2024-02-12T16:29:25.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:24.272801+0000 mon.smithi104 (mon.0) 92 : audit [DBG] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-12T16:29:25.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:24.273405+0000 mon.smithi104 (mon.0) 93 : audit [DBG] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mgr metadata", "who": "smithi104.ohwbhf", "id": "smithi104.ohwbhf"}]: dispatch 2024-02-12T16:29:25.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:24.274075+0000 mon.smithi104 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T16:29:25.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:24.274281+0000 mon.smithi104 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T16:29:25.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:24.274453+0000 mon.smithi104 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T16:29:25.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:24.287530+0000 mon.smithi104 (mon.0) 97 : cluster [INF] Manager daemon smithi104.ohwbhf is now available 2024-02-12T16:29:25.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:24.383041+0000 mon.smithi104 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:29:25.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:24.385463+0000 mon.smithi104 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:29:25.621 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:24.400454+0000 mon.smithi104 (mon.0) 100 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:29:25.621 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:24.408071+0000 mon.smithi104 (mon.0) 101 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:29:25.867 INFO:teuthology.orchestra.run.smithi104.stderr:mgr epoch 13 is available 2024-02-12T16:29:25.867 INFO:teuthology.orchestra.run.smithi104.stderr:Generating a dashboard self-signed certificate... 2024-02-12T16:29:25.867 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-12T16:29:26.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:25.277252+0000 mgr.smithi104.ohwbhf (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-12T16:29:26.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:25.277635+0000 mon.smithi104 (mon.0) 102 : cluster [DBG] mgrmap e15: smithi104.ohwbhf(active, since 1.06028s) 2024-02-12T16:29:26.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:25.283357+0000 mgr.smithi104.ohwbhf (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.465+0000 7f8280a80700 1 Processor -- start 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.465+0000 7f8280a80700 1 -- start start 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.465+0000 7f8280a80700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.465+0000 7f8280a80700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0fef80 con 0x7f827c0703f0 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.466+0000 7f827a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.466+0000 7f827a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59534/0 (socket says 172.21.15.104:59534) 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.466+0000 7f827a59c700 1 -- 172.21.15.104:0/344399000 learned_addr learned my addr 172.21.15.104:0/344399000 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.466+0000 7f827a59c700 1 -- 172.21.15.104:0/344399000 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c0ff0c0 con 0x7f827c0703f0 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.466+0000 7f827a59c700 1 --2- 172.21.15.104:0/344399000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fea90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f827000b5f0 tx=0x7f8270006ee0).ready entity=mon.0 client_cookie=eeda4377bba16e2e server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.466+0000 7f827959a700 1 -- 172.21.15.104:0/344399000 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f827000d600 con 0x7f827c0703f0 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.466+0000 7f827959a700 1 -- 172.21.15.104:0/344399000 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f827000dbf0 con 0x7f827c0703f0 2024-02-12T16:29:26.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.466+0000 7f827959a700 1 -- 172.21.15.104:0/344399000 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8270011ce0 con 0x7f827c0703f0 2024-02-12T16:29:26.472 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.467+0000 7f8280a80700 1 -- 172.21.15.104:0/344399000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 msgr2=0x7f827c0fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:26.472 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.467+0000 7f8280a80700 1 --2- 172.21.15.104:0/344399000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fea90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f827000b5f0 tx=0x7f8270006ee0).stop 2024-02-12T16:29:26.472 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.467+0000 7f8280a80700 1 -- 172.21.15.104:0/344399000 shutdown_connections 2024-02-12T16:29:26.472 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.467+0000 7f8280a80700 1 --2- 172.21.15.104:0/344399000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fea90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:26.472 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.467+0000 7f8280a80700 1 -- 172.21.15.104:0/344399000 >> 172.21.15.104:0/344399000 conn(0x7f827c0f6530 msgr2=0x7f827c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:26.472 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.467+0000 7f8280a80700 1 -- 172.21.15.104:0/344399000 shutdown_connections 2024-02-12T16:29:26.472 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.467+0000 7f8280a80700 1 -- 172.21.15.104:0/344399000 wait complete. 2024-02-12T16:29:26.472 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.468+0000 7f8280a80700 1 Processor -- start 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.468+0000 7f8280a80700 1 -- start start 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.468+0000 7f8280a80700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.468+0000 7f8280a80700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0fa990 con 0x7f827c0703f0 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.468+0000 7f827a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.468+0000 7f827a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59540/0 (socket says 172.21.15.104:59540) 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.468+0000 7f827a59c700 1 -- 172.21.15.104:0/1723557550 learned_addr learned my addr 172.21.15.104:0/1723557550 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.468+0000 7f827a59c700 1 -- 172.21.15.104:0/1723557550 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827000b200 con 0x7f827c0703f0 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.468+0000 7f827a59c700 1 --2- 172.21.15.104:0/1723557550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fc510 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f82700115c0 tx=0x7f8270006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.468+0000 7f826affd700 1 -- 172.21.15.104:0/1723557550 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f827001b410 con 0x7f827c0703f0 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.468+0000 7f826affd700 1 -- 172.21.15.104:0/1723557550 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f827001ba00 con 0x7f827c0703f0 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.469+0000 7f8280a80700 1 -- 172.21.15.104:0/1723557550 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827c0fab30 con 0x7f827c0703f0 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.469+0000 7f826affd700 1 -- 172.21.15.104:0/1723557550 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f827001aaa0 con 0x7f827c0703f0 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.469+0000 7f8280a80700 1 -- 172.21.15.104:0/1723557550 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827c0fafc0 con 0x7f827c0703f0 2024-02-12T16:29:26.473 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.469+0000 7f826affd700 1 -- 172.21.15.104:0/1723557550 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f827000f070 con 0x7f827c0703f0 2024-02-12T16:29:26.474 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.470+0000 7f826affd700 1 --2- 172.21.15.104:0/1723557550 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f826002ff70 0x7f8260032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:26.474 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.470+0000 7f826affd700 1 -- 172.21.15.104:0/1723557550 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8270041250 con 0x7f827c0703f0 2024-02-12T16:29:26.474 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.470+0000 7f8280a80700 1 -- 172.21.15.104:0/1723557550 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827c045ad0 con 0x7f827c0703f0 2024-02-12T16:29:26.474 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.470+0000 7f8279d9b700 1 --2- 172.21.15.104:0/1723557550 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f826002ff70 0x7f8260032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:26.474 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.470+0000 7f8279d9b700 1 --2- 172.21.15.104:0/1723557550 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f826002ff70 0x7f8260032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f826c0096a0 tx=0x7f826c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:26.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.474+0000 7f826affd700 1 -- 172.21.15.104:0/1723557550 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8270016070 con 0x7f827c0703f0 2024-02-12T16:29:26.631 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.629+0000 7f8280a80700 1 -- 172.21.15.104:0/1723557550 --> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f827c104b70 con 0x7f826002ff70 2024-02-12T16:29:26.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.759+0000 7f826affd700 1 -- 172.21.15.104:0/1723557550 <== mgr.14162 v2:172.21.15.104:6800/4169661527 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f827c104b70 con 0x7f826002ff70 2024-02-12T16:29:26.761 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-12T16:29:26.763 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.760+0000 7f8280a80700 1 -- 172.21.15.104:0/1723557550 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f826002ff70 msgr2=0x7f8260032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:26.763 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.760+0000 7f8280a80700 1 --2- 172.21.15.104:0/1723557550 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f826002ff70 0x7f8260032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f826c0096a0 tx=0x7f826c006b40).stop 2024-02-12T16:29:26.764 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.760+0000 7f8280a80700 1 -- 172.21.15.104:0/1723557550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 msgr2=0x7f827c0fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:26.764 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.760+0000 7f8280a80700 1 --2- 172.21.15.104:0/1723557550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fc510 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f82700115c0 tx=0x7f8270006ee0).stop 2024-02-12T16:29:26.764 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.761+0000 7f8280a80700 1 -- 172.21.15.104:0/1723557550 shutdown_connections 2024-02-12T16:29:26.764 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.761+0000 7f8280a80700 1 --2- 172.21.15.104:0/1723557550 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f826002ff70 0x7f8260032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:26.764 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.761+0000 7f8280a80700 1 --2- 172.21.15.104:0/1723557550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f827c0703f0 0x7f827c0fc510 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:26.764 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.761+0000 7f8280a80700 1 -- 172.21.15.104:0/1723557550 >> 172.21.15.104:0/1723557550 conn(0x7f827c0f6530 msgr2=0x7f827c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:26.764 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.761+0000 7f8280a80700 1 -- 172.21.15.104:0/1723557550 shutdown_connections 2024-02-12T16:29:26.764 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:26.761+0000 7f8280a80700 1 -- 172.21.15.104:0/1723557550 wait complete. 2024-02-12T16:29:27.590 INFO:teuthology.orchestra.run.smithi104.stderr:Creating initial admin user... 2024-02-12T16:29:27.591 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpwiol5woa:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-12T16:29:27.593 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:26.279906+0000 mon.smithi104 (mon.0) 103 : cluster [DBG] mgrmap e16: smithi104.ohwbhf(active, since 2s) 2024-02-12T16:29:27.593 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:26.630902+0000 mgr.smithi104.ohwbhf (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:29:27.593 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:26.757425+0000 mon.smithi104 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:27.593 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:26.760007+0000 mon.smithi104 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:28.128 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.121+0000 7ff137ae6700 1 Processor -- start 2024-02-12T16:29:28.128 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.121+0000 7ff137ae6700 1 -- start start 2024-02-12T16:29:28.128 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.121+0000 7ff137ae6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff1300fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:28.128 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.121+0000 7ff137ae6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1300fef40 con 0x7ff130070430 2024-02-12T16:29:28.128 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.121+0000 7ff135882700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff1300fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:28.128 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.121+0000 7ff135882700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff1300fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59548/0 (socket says 172.21.15.104:59548) 2024-02-12T16:29:28.128 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.121+0000 7ff135882700 1 -- 172.21.15.104:0/2744634751 learned_addr learned my addr 172.21.15.104:0/2744634751 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:28.128 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.122+0000 7ff135882700 1 -- 172.21.15.104:0/2744634751 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1300ff080 con 0x7ff130070430 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.122+0000 7ff135882700 1 --2- 172.21.15.104:0/2744634751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff1300fea50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7ff12c00b5f0 tx=0x7ff12c006ee0).ready entity=mon.0 client_cookie=6a3cd513d6a14e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.122+0000 7ff134880700 1 -- 172.21.15.104:0/2744634751 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff12c00d600 con 0x7ff130070430 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.122+0000 7ff134880700 1 -- 172.21.15.104:0/2744634751 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff12c00dbf0 con 0x7ff130070430 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.122+0000 7ff134880700 1 -- 172.21.15.104:0/2744634751 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff12c011cb0 con 0x7ff130070430 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.122+0000 7ff137ae6700 1 -- 172.21.15.104:0/2744634751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 msgr2=0x7ff1300fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.122+0000 7ff137ae6700 1 --2- 172.21.15.104:0/2744634751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff1300fea50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7ff12c00b5f0 tx=0x7ff12c006ee0).stop 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.122+0000 7ff137ae6700 1 -- 172.21.15.104:0/2744634751 shutdown_connections 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.122+0000 7ff137ae6700 1 --2- 172.21.15.104:0/2744634751 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff1300fea50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.122+0000 7ff137ae6700 1 -- 172.21.15.104:0/2744634751 >> 172.21.15.104:0/2744634751 conn(0x7ff1300f6530 msgr2=0x7ff1300f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.123+0000 7ff137ae6700 1 -- 172.21.15.104:0/2744634751 shutdown_connections 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.123+0000 7ff137ae6700 1 -- 172.21.15.104:0/2744634751 wait complete. 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.123+0000 7ff137ae6700 1 Processor -- start 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.123+0000 7ff137ae6700 1 -- start start 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.123+0000 7ff137ae6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff13010ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:28.129 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.123+0000 7ff137ae6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13010b050 con 0x7ff130070430 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.123+0000 7ff135882700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff13010ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.123+0000 7ff135882700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff13010ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59564/0 (socket says 172.21.15.104:59564) 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.123+0000 7ff135882700 1 -- 172.21.15.104:0/1856795984 learned_addr learned my addr 172.21.15.104:0/1856795984 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.124+0000 7ff135882700 1 -- 172.21.15.104:0/1856795984 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c00b200 con 0x7ff130070430 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.124+0000 7ff135882700 1 --2- 172.21.15.104:0/1856795984 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff13010ab60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff12c006800 tx=0x7ff12c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.124+0000 7ff11e7fc700 1 -- 172.21.15.104:0/1856795984 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff12c01b570 con 0x7ff130070430 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.124+0000 7ff11e7fc700 1 -- 172.21.15.104:0/1856795984 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff12c01bb60 con 0x7ff130070430 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.124+0000 7ff137ae6700 1 -- 172.21.15.104:0/1856795984 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13010b250 con 0x7ff130070430 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.124+0000 7ff11e7fc700 1 -- 172.21.15.104:0/1856795984 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff12c01a9f0 con 0x7ff130070430 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.124+0000 7ff137ae6700 1 -- 172.21.15.104:0/1856795984 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13010b6e0 con 0x7ff130070430 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.125+0000 7ff11e7fc700 1 -- 172.21.15.104:0/1856795984 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ff12c00f070 con 0x7ff130070430 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.125+0000 7ff11e7fc700 1 --2- 172.21.15.104:0/1856795984 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff11802ff70 0x7ff118032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.125+0000 7ff11e7fc700 1 -- 172.21.15.104:0/1856795984 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff12c041790 con 0x7ff130070430 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.125+0000 7ff137ae6700 1 -- 172.21.15.104:0/1856795984 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff130045ad0 con 0x7ff130070430 2024-02-12T16:29:28.130 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.125+0000 7ff135081700 1 --2- 172.21.15.104:0/1856795984 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff11802ff70 0x7ff118032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:28.131 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.125+0000 7ff135081700 1 --2- 172.21.15.104:0/1856795984 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff11802ff70 0x7ff118032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff1200096a0 tx=0x7ff120006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:28.131 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.129+0000 7ff11e7fc700 1 -- 172.21.15.104:0/1856795984 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff12c016070 con 0x7ff130070430 2024-02-12T16:29:28.296 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.293+0000 7ff137ae6700 1 -- 172.21.15.104:0/1856795984 --> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff1300ffdc0 con 0x7ff11802ff70 2024-02-12T16:29:28.512 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.509+0000 7ff11e7fc700 1 -- 172.21.15.104:0/1856795984 <== mgr.14162 v2:172.21.15.104:6800/4169661527 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7ff1300ffdc0 con 0x7ff11802ff70 2024-02-12T16:29:28.512 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$htgL3eVZmdT/M.UJpRhB.eFkWNXBB.fE7W.o4hxxfkZFzFD24TWUW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707755368, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-12T16:29:28.514 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.511+0000 7ff137ae6700 1 -- 172.21.15.104:0/1856795984 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff11802ff70 msgr2=0x7ff118032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:28.514 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.511+0000 7ff137ae6700 1 --2- 172.21.15.104:0/1856795984 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff11802ff70 0x7ff118032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff1200096a0 tx=0x7ff120006b40).stop 2024-02-12T16:29:28.514 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.511+0000 7ff137ae6700 1 -- 172.21.15.104:0/1856795984 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 msgr2=0x7ff13010ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:28.514 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.511+0000 7ff137ae6700 1 --2- 172.21.15.104:0/1856795984 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff13010ab60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff12c006800 tx=0x7ff12c006ee0).stop 2024-02-12T16:29:28.514 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.511+0000 7ff137ae6700 1 -- 172.21.15.104:0/1856795984 shutdown_connections 2024-02-12T16:29:28.514 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.511+0000 7ff137ae6700 1 --2- 172.21.15.104:0/1856795984 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff11802ff70 0x7ff118032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:28.514 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.511+0000 7ff137ae6700 1 --2- 172.21.15.104:0/1856795984 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff130070430 0x7ff13010ab60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:28.514 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.511+0000 7ff137ae6700 1 -- 172.21.15.104:0/1856795984 >> 172.21.15.104:0/1856795984 conn(0x7ff1300f6530 msgr2=0x7ff1300f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:28.515 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.511+0000 7ff137ae6700 1 -- 172.21.15.104:0/1856795984 shutdown_connections 2024-02-12T16:29:28.515 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:28.511+0000 7ff137ae6700 1 -- 172.21.15.104:0/1856795984 wait complete. 2024-02-12T16:29:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:Fetching dashboard port number... 2024-02-12T16:29:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpb1lb7pja:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxp6d3jnw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-12T16:29:29.693 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.683+0000 7f1b5d78b700 1 Processor -- start 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.683+0000 7f1b5d78b700 1 -- start start 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.683+0000 7f1b5d78b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b580fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.683+0000 7f1b5d78b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580fedd0 con 0x7f1b580703f0 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.683+0000 7f1b56ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b580fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.684+0000 7f1b56ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b580fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59570/0 (socket says 172.21.15.104:59570) 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.684+0000 7f1b56ffd700 1 -- 172.21.15.104:0/374531704 learned_addr learned my addr 172.21.15.104:0/374531704 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.684+0000 7f1b56ffd700 1 -- 172.21.15.104:0/374531704 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b580fef10 con 0x7f1b580703f0 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.684+0000 7f1b56ffd700 1 --2- 172.21.15.104:0/374531704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b580fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1b4c00b5f0 tx=0x7f1b4c006ee0).ready entity=mon.0 client_cookie=895616b3c082c2cd server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.684+0000 7f1b55ffb700 1 -- 172.21.15.104:0/374531704 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b4c00d600 con 0x7f1b580703f0 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.684+0000 7f1b55ffb700 1 -- 172.21.15.104:0/374531704 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1b4c00dbf0 con 0x7f1b580703f0 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.684+0000 7f1b55ffb700 1 -- 172.21.15.104:0/374531704 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b4c011ce0 con 0x7f1b580703f0 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.684+0000 7f1b5d78b700 1 -- 172.21.15.104:0/374531704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 msgr2=0x7f1b580fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.684+0000 7f1b5d78b700 1 --2- 172.21.15.104:0/374531704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b580fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1b4c00b5f0 tx=0x7f1b4c006ee0).stop 2024-02-12T16:29:29.694 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.685+0000 7f1b5d78b700 1 -- 172.21.15.104:0/374531704 shutdown_connections 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.685+0000 7f1b5d78b700 1 --2- 172.21.15.104:0/374531704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b580fe8e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.685+0000 7f1b5d78b700 1 -- 172.21.15.104:0/374531704 >> 172.21.15.104:0/374531704 conn(0x7f1b580f63c0 msgr2=0x7f1b580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.685+0000 7f1b5d78b700 1 -- 172.21.15.104:0/374531704 shutdown_connections 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.685+0000 7f1b5d78b700 1 -- 172.21.15.104:0/374531704 wait complete. 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.685+0000 7f1b5d78b700 1 Processor -- start 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.685+0000 7f1b5d78b700 1 -- start start 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.685+0000 7f1b5d78b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b5810a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.685+0000 7f1b5d78b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5810ae80 con 0x7f1b580703f0 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.686+0000 7f1b56ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b5810a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.686+0000 7f1b56ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b5810a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59580/0 (socket says 172.21.15.104:59580) 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.686+0000 7f1b56ffd700 1 -- 172.21.15.104:0/632864093 learned_addr learned my addr 172.21.15.104:0/632864093 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.686+0000 7f1b56ffd700 1 -- 172.21.15.104:0/632864093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c00b200 con 0x7f1b580703f0 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.686+0000 7f1b56ffd700 1 --2- 172.21.15.104:0/632864093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b5810a990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1b4c011ed0 tx=0x7f1b4c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:29.695 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.686+0000 7f1b3f7fe700 1 -- 172.21.15.104:0/632864093 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b4c01b570 con 0x7f1b580703f0 2024-02-12T16:29:29.696 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.686+0000 7f1b3f7fe700 1 -- 172.21.15.104:0/632864093 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1b4c01bb60 con 0x7f1b580703f0 2024-02-12T16:29:29.696 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.686+0000 7f1b5d78b700 1 -- 172.21.15.104:0/632864093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5810b080 con 0x7f1b580703f0 2024-02-12T16:29:29.696 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.686+0000 7f1b3f7fe700 1 -- 172.21.15.104:0/632864093 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b4c01aad0 con 0x7f1b580703f0 2024-02-12T16:29:29.696 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.686+0000 7f1b5d78b700 1 -- 172.21.15.104:0/632864093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5810b510 con 0x7f1b580703f0 2024-02-12T16:29:29.696 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.687+0000 7f1b5d78b700 1 -- 172.21.15.104:0/632864093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b58045ad0 con 0x7f1b580703f0 2024-02-12T16:29:29.696 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.688+0000 7f1b3f7fe700 1 -- 172.21.15.104:0/632864093 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f1b4c00f070 con 0x7f1b580703f0 2024-02-12T16:29:29.696 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.688+0000 7f1b3f7fe700 1 --2- 172.21.15.104:0/632864093 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b4402ff70 0x7f1b44032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:29.696 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.688+0000 7f1b3f7fe700 1 -- 172.21.15.104:0/632864093 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b4c041640 con 0x7f1b580703f0 2024-02-12T16:29:29.696 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.688+0000 7f1b567fc700 1 --2- 172.21.15.104:0/632864093 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b4402ff70 0x7f1b44032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:29.696 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.688+0000 7f1b567fc700 1 --2- 172.21.15.104:0/632864093 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b4402ff70 0x7f1b44032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f1b480096a0 tx=0x7f1b48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:29.697 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.692+0000 7f1b3f7fe700 1 -- 172.21.15.104:0/632864093 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b4c022310 con 0x7f1b580703f0 2024-02-12T16:29:29.844 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.842+0000 7f1b5d78b700 1 -- 172.21.15.104:0/632864093 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f1b580ffc50 con 0x7f1b580703f0 2024-02-12T16:29:29.845 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.842+0000 7f1b3f7fe700 1 -- 172.21.15.104:0/632864093 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f1b4c016070 con 0x7f1b580703f0 2024-02-12T16:29:29.845 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 8443 2024-02-12T16:29:29.847 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.844+0000 7f1b5d78b700 1 -- 172.21.15.104:0/632864093 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b4402ff70 msgr2=0x7f1b44032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:29.847 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.844+0000 7f1b5d78b700 1 --2- 172.21.15.104:0/632864093 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b4402ff70 0x7f1b44032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f1b480096a0 tx=0x7f1b48006b40).stop 2024-02-12T16:29:29.847 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.844+0000 7f1b5d78b700 1 -- 172.21.15.104:0/632864093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 msgr2=0x7f1b5810a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:29.847 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.844+0000 7f1b5d78b700 1 --2- 172.21.15.104:0/632864093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b5810a990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1b4c011ed0 tx=0x7f1b4c006ee0).stop 2024-02-12T16:29:29.847 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.844+0000 7f1b5d78b700 1 -- 172.21.15.104:0/632864093 shutdown_connections 2024-02-12T16:29:29.847 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.844+0000 7f1b5d78b700 1 --2- 172.21.15.104:0/632864093 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b4402ff70 0x7f1b44032430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:29.847 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.844+0000 7f1b5d78b700 1 --2- 172.21.15.104:0/632864093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b580703f0 0x7f1b5810a990 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:29.847 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.844+0000 7f1b5d78b700 1 -- 172.21.15.104:0/632864093 >> 172.21.15.104:0/632864093 conn(0x7f1b580f63c0 msgr2=0x7f1b580f7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:29.847 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.845+0000 7f1b5d78b700 1 -- 172.21.15.104:0/632864093 shutdown_connections 2024-02-12T16:29:29.847 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-02-12T16:29:29.845+0000 7f1b5d78b700 1 -- 172.21.15.104:0/632864093 wait complete. 2024-02-12T16:29:29.869 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:29 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:28.295346+0000 mgr.smithi104.ohwbhf (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:29:29.869 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:29 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:28.510574+0000 mon.smithi104 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:30.361 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-12T16:29:30.370 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: disabled 2024-02-12T16:29:30.371 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: systemctl is-active firewalld.service 2024-02-12T16:29:30.378 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-02-12T16:29:30.378 INFO:teuthology.orchestra.run.smithi104.stderr:firewalld.service is not enabled 2024-02-12T16:29:30.378 INFO:teuthology.orchestra.run.smithi104.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-12T16:29:30.381 INFO:teuthology.orchestra.run.smithi104.stderr:Ceph Dashboard is now available at: 2024-02-12T16:29:30.381 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-02-12T16:29:30.381 INFO:teuthology.orchestra.run.smithi104.stderr: URL: https://smithi104.front.sepia.ceph.com:8443/ 2024-02-12T16:29:30.381 INFO:teuthology.orchestra.run.smithi104.stderr: User: admin 2024-02-12T16:29:30.381 INFO:teuthology.orchestra.run.smithi104.stderr: Password: gaavrpwvjv 2024-02-12T16:29:30.381 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-02-12T16:29:30.382 INFO:teuthology.orchestra.run.smithi104.stderr:You can access the Ceph CLI with: 2024-02-12T16:29:30.382 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-02-12T16:29:30.382 INFO:teuthology.orchestra.run.smithi104.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:29:30.382 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-02-12T16:29:30.382 INFO:teuthology.orchestra.run.smithi104.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-12T16:29:30.382 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-02-12T16:29:30.382 INFO:teuthology.orchestra.run.smithi104.stderr: ceph telemetry on 2024-02-12T16:29:30.382 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-02-12T16:29:30.383 INFO:teuthology.orchestra.run.smithi104.stderr:For more information see: 2024-02-12T16:29:30.383 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-02-12T16:29:30.383 INFO:teuthology.orchestra.run.smithi104.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-12T16:29:30.383 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-02-12T16:29:30.383 INFO:teuthology.orchestra.run.smithi104.stderr:Bootstrap complete. 2024-02-12T16:29:30.383 INFO:teuthology.orchestra.run.smithi104.stderr:Releasing lock 139836925118952 on /run/cephadm/aa0fd796-c9c3-11ee-95b9-87774f69a715.lock 2024-02-12T16:29:30.383 INFO:teuthology.orchestra.run.smithi104.stderr:Lock 139836925118952 released on /run/cephadm/aa0fd796-c9c3-11ee-95b9-87774f69a715.lock 2024-02-12T16:29:30.415 INFO:tasks.cephadm:Fetching config... 2024-02-12T16:29:30.415 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-12T16:29:30.415 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-12T16:29:30.473 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-12T16:29:30.473 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-12T16:29:30.473 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-12T16:29:30.531 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-12T16:29:30.532 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-12T16:29:30.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/mon.smithi104/keyring of=/dev/stdout 2024-02-12T16:29:30.599 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-12T16:29:30.599 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-12T16:29:30.599 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-12T16:29:30.657 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-12T16:29:30.657 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJf7a0S5c9C7Mq6QgVacJq3Uis8FgkK7Z3ogsBkGGxLJOMKz4zVB+j3F6zYRChcr0BQ+3PZVZuSofNO3AYC6DPEfTsHwLnUCkQtSds5By+VaM4KeAJCykmxVrH5K88staZMpbnXtWPdjgFfsNrDuZtYHPz3SqTkdKjqgFz6uaiJqhejjmIwlbBvuADuAktoH+zMdQWRCyup+cIlEm8GL8fvXlYyM0gKhTbfvpC9trY3H8TddI7PoVlJgyaqMOO9HrabO3Nw6cHTMLbAHJNJInQUxXj4xPiBtm8blUnjE+qr5RFn5TXt3dukgVtUVu3dWnEtHFQ7VgmwbGpnuDvoYLIBr3WtKOOM+0DLKSlzXhmhKayZkZJoDk7nE0Tk2SWJXwGdLCrDYKD+m0dbkmrraGjuAX+lnyci7shfYS6AYkxQpjmTaWp7t72Q8OBPlqsSpsyEX0MjgshbDofh4xW/XiCdfMiocZcoGukPzyjWNUezlMjuJ3suDNJx0sRH4+iwIc= ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-12T16:29:30.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:29.843688+0000 mon.smithi104 (mon.0) 107 : audit [DBG] from='client.? 172.21.15.104:0/632864093' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-12T16:29:30.709 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJf7a0S5c9C7Mq6QgVacJq3Uis8FgkK7Z3ogsBkGGxLJOMKz4zVB+j3F6zYRChcr0BQ+3PZVZuSofNO3AYC6DPEfTsHwLnUCkQtSds5By+VaM4KeAJCykmxVrH5K88staZMpbnXtWPdjgFfsNrDuZtYHPz3SqTkdKjqgFz6uaiJqhejjmIwlbBvuADuAktoH+zMdQWRCyup+cIlEm8GL8fvXlYyM0gKhTbfvpC9trY3H8TddI7PoVlJgyaqMOO9HrabO3Nw6cHTMLbAHJNJInQUxXj4xPiBtm8blUnjE+qr5RFn5TXt3dukgVtUVu3dWnEtHFQ7VgmwbGpnuDvoYLIBr3WtKOOM+0DLKSlzXhmhKayZkZJoDk7nE0Tk2SWJXwGdLCrDYKD+m0dbkmrraGjuAX+lnyci7shfYS6AYkxQpjmTaWp7t72Q8OBPlqsSpsyEX0MjgshbDofh4xW/XiCdfMiocZcoGukPzyjWNUezlMjuJ3suDNJx0sRH4+iwIc= ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:29:30.720 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJf7a0S5c9C7Mq6QgVacJq3Uis8FgkK7Z3ogsBkGGxLJOMKz4zVB+j3F6zYRChcr0BQ+3PZVZuSofNO3AYC6DPEfTsHwLnUCkQtSds5By+VaM4KeAJCykmxVrH5K88staZMpbnXtWPdjgFfsNrDuZtYHPz3SqTkdKjqgFz6uaiJqhejjmIwlbBvuADuAktoH+zMdQWRCyup+cIlEm8GL8fvXlYyM0gKhTbfvpC9trY3H8TddI7PoVlJgyaqMOO9HrabO3Nw6cHTMLbAHJNJInQUxXj4xPiBtm8blUnjE+qr5RFn5TXt3dukgVtUVu3dWnEtHFQ7VgmwbGpnuDvoYLIBr3WtKOOM+0DLKSlzXhmhKayZkZJoDk7nE0Tk2SWJXwGdLCrDYKD+m0dbkmrraGjuAX+lnyci7shfYS6AYkxQpjmTaWp7t72Q8OBPlqsSpsyEX0MjgshbDofh4xW/XiCdfMiocZcoGukPzyjWNUezlMjuJ3suDNJx0sRH4+iwIc= ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-12T16:29:30.765 INFO:teuthology.orchestra.run.smithi149.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJf7a0S5c9C7Mq6QgVacJq3Uis8FgkK7Z3ogsBkGGxLJOMKz4zVB+j3F6zYRChcr0BQ+3PZVZuSofNO3AYC6DPEfTsHwLnUCkQtSds5By+VaM4KeAJCykmxVrH5K88staZMpbnXtWPdjgFfsNrDuZtYHPz3SqTkdKjqgFz6uaiJqhejjmIwlbBvuADuAktoH+zMdQWRCyup+cIlEm8GL8fvXlYyM0gKhTbfvpC9trY3H8TddI7PoVlJgyaqMOO9HrabO3Nw6cHTMLbAHJNJInQUxXj4xPiBtm8blUnjE+qr5RFn5TXt3dukgVtUVu3dWnEtHFQ7VgmwbGpnuDvoYLIBr3WtKOOM+0DLKSlzXhmhKayZkZJoDk7nE0Tk2SWJXwGdLCrDYKD+m0dbkmrraGjuAX+lnyci7shfYS6AYkxQpjmTaWp7t72Q8OBPlqsSpsyEX0MjgshbDofh4xW/XiCdfMiocZcoGukPzyjWNUezlMjuJ3suDNJx0sRH4+iwIc= ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715 2024-02-12T16:29:30.777 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-12T16:29:31.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:31 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:30.512514+0000 mon.smithi104 (mon.0) 108 : cluster [DBG] mgrmap e17: smithi104.ohwbhf(active, since 6s) 2024-02-12T16:29:33.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.088+0000 7efed856e700 1 -- 172.21.15.104:0/1982426983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efed0060280 msgr2=0x7efed00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:33.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.088+0000 7efed856e700 1 --2- 172.21.15.104:0/1982426983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efed0060280 0x7efed00f90f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7efecc009fb0 tx=0x7efecc009b30).stop 2024-02-12T16:29:33.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.088+0000 7efed856e700 1 -- 172.21.15.104:0/1982426983 shutdown_connections 2024-02-12T16:29:33.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.088+0000 7efed856e700 1 --2- 172.21.15.104:0/1982426983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efed0060280 0x7efed00f90f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:33.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.088+0000 7efed856e700 1 -- 172.21.15.104:0/1982426983 >> 172.21.15.104:0/1982426983 conn(0x7efed00f0b70 msgr2=0x7efed00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:33.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.088+0000 7efed856e700 1 -- 172.21.15.104:0/1982426983 shutdown_connections 2024-02-12T16:29:33.434 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.088+0000 7efed856e700 1 -- 172.21.15.104:0/1982426983 wait complete. 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.088+0000 7efed856e700 1 Processor -- start 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.088+0000 7efed856e700 1 -- start start 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efed856e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efed0060280 0x7efed0106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efed856e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed0106c40 con 0x7efed0060280 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efed630a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efed0060280 0x7efed0106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efed630a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efed0060280 0x7efed0106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:42006/0 (socket says 172.21.15.104:42006) 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efed630a700 1 -- 172.21.15.104:0/3568665650 learned_addr learned my addr 172.21.15.104:0/3568665650 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efed630a700 1 -- 172.21.15.104:0/3568665650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc005040 con 0x7efed0060280 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efed630a700 1 --2- 172.21.15.104:0/3568665650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efed0060280 0x7efed0106750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7efecc0089f0 tx=0x7efecc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efec6ffd700 1 -- 172.21.15.104:0/3568665650 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efecc009320 con 0x7efed0060280 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efec6ffd700 1 -- 172.21.15.104:0/3568665650 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efecc0136f0 con 0x7efed0060280 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efed856e700 1 -- 172.21.15.104:0/3568665650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed0106e40 con 0x7efed0060280 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efec6ffd700 1 -- 172.21.15.104:0/3568665650 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efecc014420 con 0x7efed0060280 2024-02-12T16:29:33.435 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.089+0000 7efed856e700 1 -- 172.21.15.104:0/3568665650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed01072d0 con 0x7efed0060280 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.090+0000 7efec6ffd700 1 -- 172.21.15.104:0/3568665650 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efecc014580 con 0x7efed0060280 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.090+0000 7efec6ffd700 1 --2- 172.21.15.104:0/3568665650 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7efebc02ffb0 0x7efebc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.090+0000 7efec6ffd700 1 -- 172.21.15.104:0/3568665650 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efecc040ec0 con 0x7efed0060280 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.090+0000 7efed5b09700 1 --2- 172.21.15.104:0/3568665650 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7efebc02ffb0 0x7efebc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.090+0000 7efed856e700 1 -- 172.21.15.104:0/3568665650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed0059100 con 0x7efed0060280 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.090+0000 7efed5b09700 1 --2- 172.21.15.104:0/3568665650 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7efebc02ffb0 0x7efebc032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7efec00096a0 tx=0x7efec0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.094+0000 7efec6ffd700 1 -- 172.21.15.104:0/3568665650 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efecc0273f0 con 0x7efed0060280 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.244+0000 7efed856e700 1 -- 172.21.15.104:0/3568665650 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7efed0045ad0 con 0x7efed0060280 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.247+0000 7efec6ffd700 1 -- 172.21.15.104:0/3568665650 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7efecc01a230 con 0x7efed0060280 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.249+0000 7efed856e700 1 -- 172.21.15.104:0/3568665650 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7efebc02ffb0 msgr2=0x7efebc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.249+0000 7efed856e700 1 --2- 172.21.15.104:0/3568665650 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7efebc02ffb0 0x7efebc032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7efec00096a0 tx=0x7efec0006b40).stop 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.249+0000 7efed856e700 1 -- 172.21.15.104:0/3568665650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efed0060280 msgr2=0x7efed0106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:29:33.436 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.249+0000 7efed856e700 1 --2- 172.21.15.104:0/3568665650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efed0060280 0x7efed0106750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7efecc0089f0 tx=0x7efecc00b040).stop 2024-02-12T16:29:33.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.249+0000 7efed856e700 1 -- 172.21.15.104:0/3568665650 shutdown_connections 2024-02-12T16:29:33.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.249+0000 7efed856e700 1 --2- 172.21.15.104:0/3568665650 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7efebc02ffb0 0x7efebc032470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:33.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.249+0000 7efed856e700 1 --2- 172.21.15.104:0/3568665650 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efed0060280 0x7efed0106750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:29:33.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.249+0000 7efed856e700 1 -- 172.21.15.104:0/3568665650 >> 172.21.15.104:0/3568665650 conn(0x7efed00f0b70 msgr2=0x7efed00f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:29:33.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.249+0000 7efed856e700 1 -- 172.21.15.104:0/3568665650 shutdown_connections 2024-02-12T16:29:33.437 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:29:33.249+0000 7efed856e700 1 -- 172.21.15.104:0/3568665650 wait complete. 2024-02-12T16:29:34.195 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi149 2024-02-12T16:29:34.195 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-12T16:29:34.195 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/etc/ceph/ceph.conf 2024-02-12T16:29:34.217 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-02-12T16:29:34.217 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-12T16:29:34.277 INFO:tasks.cephadm:Adding host smithi149 to orchestrator... 2024-02-12T16:29:34.277 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph orch host add smithi149 2024-02-12T16:29:34.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:33.248512+0000 mon.smithi104 (mon.0) 109 : audit [INF] from='client.? 172.21.15.104:0/3568665650' entity='client.admin' 2024-02-12T16:29:35.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:35 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:34.524199+0000 mon.smithi104 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:35.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:35 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:34.742634+0000 mon.smithi104 (mon.0) 111 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:37.045 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:35.756593+0000 mon.smithi104 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:40.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:39.108298+0000 mon.smithi104 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:40.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:39.367377+0000 mon.smithi104 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:40.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:29:39.370283+0000 mon.smithi104 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:29:41.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:41 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:29:39.376687+0000 mgr.smithi104.ohwbhf (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi104 on smithi104 2024-02-12T16:29:45.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:45 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:44.275360+0000 mgr.smithi104.ohwbhf (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:29:48.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:46.275631+0000 mgr.smithi104.ohwbhf (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:29:50.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:48.275917+0000 mgr.smithi104.ohwbhf (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:29:52.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:50.276234+0000 mgr.smithi104.ohwbhf (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:29:54.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:52.276556+0000 mgr.smithi104.ohwbhf (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:29:56.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:54.276772+0000 mgr.smithi104.ohwbhf (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:29:58.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:29:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:56.277043+0000 mgr.smithi104.ohwbhf (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:00.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:29:58.277385+0000 mgr.smithi104.ohwbhf (mgr.14162) 13 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:01.756 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.752+0000 7ffbb0af2700 1 -- 172.21.15.149:0/1304653399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbac0f62a0 msgr2=0x7ffbac0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:01.757 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.752+0000 7ffbb0af2700 1 --2- 172.21.15.149:0/1304653399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbac0f62a0 0x7ffbac0f66c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7ffb9c009fb0 tx=0x7ffb9c009b30).stop 2024-02-12T16:30:01.757 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.752+0000 7ffbb0af2700 1 -- 172.21.15.149:0/1304653399 shutdown_connections 2024-02-12T16:30:01.757 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.752+0000 7ffbb0af2700 1 --2- 172.21.15.149:0/1304653399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbac0f62a0 0x7ffbac0f66c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:01.757 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.752+0000 7ffbb0af2700 1 -- 172.21.15.149:0/1304653399 >> 172.21.15.149:0/1304653399 conn(0x7ffbac0f1aa0 msgr2=0x7ffbac0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:01.757 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.752+0000 7ffbb0af2700 1 -- 172.21.15.149:0/1304653399 shutdown_connections 2024-02-12T16:30:01.757 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.752+0000 7ffbb0af2700 1 -- 172.21.15.149:0/1304653399 wait complete. 2024-02-12T16:30:01.757 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.752+0000 7ffbb0af2700 1 Processor -- start 2024-02-12T16:30:01.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.753+0000 7ffbb0af2700 1 -- start start 2024-02-12T16:30:01.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.753+0000 7ffbb0af2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbac0f62a0 0x7ffbac1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:01.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.753+0000 7ffbb0af2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac106ac0 con 0x7ffbac0f62a0 2024-02-12T16:30:01.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.753+0000 7ffbab7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbac0f62a0 0x7ffbac1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:01.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.753+0000 7ffbab7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbac0f62a0 0x7ffbac1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:47866/0 (socket says 172.21.15.149:47866) 2024-02-12T16:30:01.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.753+0000 7ffbab7fe700 1 -- 172.21.15.149:0/3468192507 learned_addr learned my addr 172.21.15.149:0/3468192507 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:01.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.753+0000 7ffbab7fe700 1 -- 172.21.15.149:0/3468192507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c005040 con 0x7ffbac0f62a0 2024-02-12T16:30:01.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.754+0000 7ffbab7fe700 1 --2- 172.21.15.149:0/3468192507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbac0f62a0 0x7ffbac1065d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7ffb9c0089f0 tx=0x7ffb9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:01.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.754+0000 7ffb93fff700 1 -- 172.21.15.149:0/3468192507 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb9c009320 con 0x7ffbac0f62a0 2024-02-12T16:30:01.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.754+0000 7ffb93fff700 1 -- 172.21.15.149:0/3468192507 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb9c013380 con 0x7ffbac0f62a0 2024-02-12T16:30:01.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.754+0000 7ffbb0af2700 1 -- 172.21.15.149:0/3468192507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbac106cc0 con 0x7ffbac0f62a0 2024-02-12T16:30:01.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.754+0000 7ffb93fff700 1 -- 172.21.15.149:0/3468192507 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb9c014e20 con 0x7ffbac0f62a0 2024-02-12T16:30:01.760 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.754+0000 7ffbb0af2700 1 -- 172.21.15.149:0/3468192507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbac107150 con 0x7ffbac0f62a0 2024-02-12T16:30:01.761 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.755+0000 7ffb93fff700 1 -- 172.21.15.149:0/3468192507 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb9c0095f0 con 0x7ffbac0f62a0 2024-02-12T16:30:01.762 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.755+0000 7ffbb0af2700 1 -- 172.21.15.149:0/3468192507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbac045ad0 con 0x7ffbac0f62a0 2024-02-12T16:30:01.762 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.755+0000 7ffb93fff700 1 --2- 172.21.15.149:0/3468192507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb9402ff10 0x7ffb940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:01.762 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.755+0000 7ffb93fff700 1 -- 172.21.15.149:0/3468192507 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb9c040e90 con 0x7ffbac0f62a0 2024-02-12T16:30:01.762 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.755+0000 7ffbaaffd700 1 --2- 172.21.15.149:0/3468192507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb9402ff10 0x7ffb940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:01.762 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.755+0000 7ffbaaffd700 1 --2- 172.21.15.149:0/3468192507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb9402ff10 0x7ffb940323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ffba00096a0 tx=0x7ffba0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:01.763 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.759+0000 7ffb93fff700 1 -- 172.21.15.149:0/3468192507 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb9c014420 con 0x7ffbac0f62a0 2024-02-12T16:30:01.933 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:01.931+0000 7ffbb0af2700 1 -- 172.21.15.149:0/3468192507 --> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi149", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbac0f2ba0 con 0x7ffb9402ff10 2024-02-12T16:30:02.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:00.277647+0000 mgr.smithi104.ohwbhf (mgr.14162) 14 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:02.531 INFO:teuthology.orchestra.run.smithi149.stdout:Added host 'smithi149' 2024-02-12T16:30:02.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:02.528+0000 7ffb93fff700 1 -- 172.21.15.149:0/3468192507 <== mgr.14162 v2:172.21.15.104:6800/4169661527 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7ffbac0f2ba0 con 0x7ffb9402ff10 2024-02-12T16:30:02.533 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:02.531+0000 7ffbb0af2700 1 -- 172.21.15.149:0/3468192507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb9402ff10 msgr2=0x7ffb940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:02.533 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:02.531+0000 7ffbb0af2700 1 --2- 172.21.15.149:0/3468192507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb9402ff10 0x7ffb940323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ffba00096a0 tx=0x7ffba0006b40).stop 2024-02-12T16:30:02.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:02.531+0000 7ffbb0af2700 1 -- 172.21.15.149:0/3468192507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbac0f62a0 msgr2=0x7ffbac1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:02.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:02.531+0000 7ffbb0af2700 1 --2- 172.21.15.149:0/3468192507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbac0f62a0 0x7ffbac1065d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7ffb9c0089f0 tx=0x7ffb9c00b040).stop 2024-02-12T16:30:02.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:02.531+0000 7ffbb0af2700 1 -- 172.21.15.149:0/3468192507 shutdown_connections 2024-02-12T16:30:02.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:02.531+0000 7ffbb0af2700 1 --2- 172.21.15.149:0/3468192507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb9402ff10 0x7ffb940323d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:02.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:02.531+0000 7ffbb0af2700 1 --2- 172.21.15.149:0/3468192507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbac0f62a0 0x7ffbac1065d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:02.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:02.531+0000 7ffbb0af2700 1 -- 172.21.15.149:0/3468192507 >> 172.21.15.149:0/3468192507 conn(0x7ffbac0f1aa0 msgr2=0x7ffbac0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:02.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:02.531+0000 7ffbb0af2700 1 -- 172.21.15.149:0/3468192507 shutdown_connections 2024-02-12T16:30:02.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:02.531+0000 7ffbb0af2700 1 -- 172.21.15.149:0/3468192507 wait complete. 2024-02-12T16:30:03.071 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-12T16:30:03.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:03 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:01.933813+0000 mgr.smithi104.ohwbhf (mgr.14162) 15 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi149", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:30:03.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:03 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:02.529119+0000 mon.smithi104 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:30:04.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:02.277950+0000 mgr.smithi104.ohwbhf (mgr.14162) 16 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:04.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:30:02.529475+0000 mgr.smithi104.ohwbhf (mgr.14162) 17 : cephadm [INF] Added host smithi149 2024-02-12T16:30:04.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.859+0000 7f1f1be54700 1 -- 172.21.15.149:0/3488232328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f140f5410 msgr2=0x7f1f140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:04.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.859+0000 7f1f1be54700 1 --2- 172.21.15.149:0/3488232328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f140f5410 0x7f1f140f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1f04009fb0 tx=0x7f1f04009b30).stop 2024-02-12T16:30:04.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.859+0000 7f1f1be54700 1 -- 172.21.15.149:0/3488232328 shutdown_connections 2024-02-12T16:30:04.863 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.859+0000 7f1f1be54700 1 --2- 172.21.15.149:0/3488232328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f140f5410 0x7f1f140f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.859+0000 7f1f1be54700 1 -- 172.21.15.149:0/3488232328 >> 172.21.15.149:0/3488232328 conn(0x7f1f140f0b70 msgr2=0x7f1f140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.859+0000 7f1f1be54700 1 -- 172.21.15.149:0/3488232328 shutdown_connections 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.859+0000 7f1f1be54700 1 -- 172.21.15.149:0/3488232328 wait complete. 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.860+0000 7f1f1be54700 1 Processor -- start 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.860+0000 7f1f1be54700 1 -- start start 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.860+0000 7f1f1be54700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f140f5410 0x7f1f141094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.860+0000 7f1f1be54700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f14060240 con 0x7f1f140f5410 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.860+0000 7f1f19bf0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f140f5410 0x7f1f141094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.860+0000 7f1f19bf0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f140f5410 0x7f1f141094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:37298/0 (socket says 172.21.15.149:37298) 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.860+0000 7f1f19bf0700 1 -- 172.21.15.149:0/2195387484 learned_addr learned my addr 172.21.15.149:0/2195387484 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.861+0000 7f1f19bf0700 1 -- 172.21.15.149:0/2195387484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f04005040 con 0x7f1f140f5410 2024-02-12T16:30:04.864 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.861+0000 7f1f19bf0700 1 --2- 172.21.15.149:0/2195387484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f140f5410 0x7f1f141094e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f1f04005010 tx=0x7f1f0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:04.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.861+0000 7f1f0a7fc700 1 -- 172.21.15.149:0/2195387484 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f04009320 con 0x7f1f140f5410 2024-02-12T16:30:04.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.861+0000 7f1f0a7fc700 1 -- 172.21.15.149:0/2195387484 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f04013380 con 0x7f1f140f5410 2024-02-12T16:30:04.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.861+0000 7f1f1be54700 1 -- 172.21.15.149:0/2195387484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f14060440 con 0x7f1f140f5410 2024-02-12T16:30:04.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.861+0000 7f1f0a7fc700 1 -- 172.21.15.149:0/2195387484 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f04014e50 con 0x7f1f140f5410 2024-02-12T16:30:04.865 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.861+0000 7f1f1be54700 1 -- 172.21.15.149:0/2195387484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f14109b90 con 0x7f1f140f5410 2024-02-12T16:30:04.866 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.862+0000 7f1f0a7fc700 1 -- 172.21.15.149:0/2195387484 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f040095f0 con 0x7f1f140f5410 2024-02-12T16:30:04.866 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.862+0000 7f1f1be54700 1 -- 172.21.15.149:0/2195387484 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f14045ad0 con 0x7f1f140f5410 2024-02-12T16:30:04.866 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.862+0000 7f1f0a7fc700 1 --2- 172.21.15.149:0/2195387484 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f0002ff10 0x7f1f000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:04.866 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.862+0000 7f1f0a7fc700 1 -- 172.21.15.149:0/2195387484 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f04040fa0 con 0x7f1f140f5410 2024-02-12T16:30:04.866 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.862+0000 7f1f193ef700 1 --2- 172.21.15.149:0/2195387484 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f0002ff10 0x7f1f000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:04.866 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.863+0000 7f1f193ef700 1 --2- 172.21.15.149:0/2195387484 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f0002ff10 0x7f1f000323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f1f100096a0 tx=0x7f1f10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:04.868 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:04.867+0000 7f1f0a7fc700 1 -- 172.21.15.149:0/2195387484 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f04014420 con 0x7f1f140f5410 2024-02-12T16:30:05.039 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.038+0000 7f1f1be54700 1 -- 172.21.15.149:0/2195387484 --> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f140f1d20 con 0x7f1f0002ff10 2024-02-12T16:30:05.040 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.039+0000 7f1f0a7fc700 1 -- 172.21.15.149:0/2195387484 <== mgr.14162 v2:172.21.15.104:6800/4169661527 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f1f140f1d20 con 0x7f1f0002ff10 2024-02-12T16:30:05.041 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:05.041 INFO:teuthology.orchestra.run.smithi149.stdout:[{"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}, {"addr": "smithi149", "hostname": "smithi149", "labels": [], "status": ""}] 2024-02-12T16:30:05.042 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.041+0000 7f1f1be54700 1 -- 172.21.15.149:0/2195387484 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f0002ff10 msgr2=0x7f1f000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:05.042 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.041+0000 7f1f1be54700 1 --2- 172.21.15.149:0/2195387484 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f0002ff10 0x7f1f000323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f1f100096a0 tx=0x7f1f10006b40).stop 2024-02-12T16:30:05.042 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.041+0000 7f1f1be54700 1 -- 172.21.15.149:0/2195387484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f140f5410 msgr2=0x7f1f141094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:05.042 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.041+0000 7f1f1be54700 1 --2- 172.21.15.149:0/2195387484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f140f5410 0x7f1f141094e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f1f04005010 tx=0x7f1f0400b040).stop 2024-02-12T16:30:05.043 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.041+0000 7f1f1be54700 1 -- 172.21.15.149:0/2195387484 shutdown_connections 2024-02-12T16:30:05.043 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.041+0000 7f1f1be54700 1 --2- 172.21.15.149:0/2195387484 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f0002ff10 0x7f1f000323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:05.043 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.041+0000 7f1f1be54700 1 --2- 172.21.15.149:0/2195387484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f140f5410 0x7f1f141094e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:05.043 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.041+0000 7f1f1be54700 1 -- 172.21.15.149:0/2195387484 >> 172.21.15.149:0/2195387484 conn(0x7f1f140f0b70 msgr2=0x7f1f140f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:05.043 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.041+0000 7f1f1be54700 1 -- 172.21.15.149:0/2195387484 shutdown_connections 2024-02-12T16:30:05.043 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:05.041+0000 7f1f1be54700 1 -- 172.21.15.149:0/2195387484 wait complete. 2024-02-12T16:30:05.678 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-12T16:30:05.678 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph osd crush tunables default 2024-02-12T16:30:06.355 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:04.278327+0000 mgr.smithi104.ohwbhf (mgr.14162) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:06.355 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:05.039175+0000 mgr.smithi104.ohwbhf (mgr.14162) 19 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:30:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.613+0000 7f75b91d6700 1 -- 172.21.15.104:0/1714087917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75b40f6510 msgr2=0x7f75b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.613+0000 7f75b91d6700 1 --2- 172.21.15.104:0/1714087917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75b40f6510 0x7f75b40f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f75a8009fb0 tx=0x7f75a8009b30).stop 2024-02-12T16:30:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.613+0000 7f75b91d6700 1 -- 172.21.15.104:0/1714087917 shutdown_connections 2024-02-12T16:30:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.613+0000 7f75b91d6700 1 --2- 172.21.15.104:0/1714087917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75b40f6510 0x7f75b40f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.613+0000 7f75b91d6700 1 -- 172.21.15.104:0/1714087917 >> 172.21.15.104:0/1714087917 conn(0x7f75b40f1c70 msgr2=0x7f75b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.613+0000 7f75b91d6700 1 -- 172.21.15.104:0/1714087917 shutdown_connections 2024-02-12T16:30:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.613+0000 7f75b91d6700 1 -- 172.21.15.104:0/1714087917 wait complete. 2024-02-12T16:30:07.651 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.613+0000 7f75b91d6700 1 Processor -- start 2024-02-12T16:30:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f75b91d6700 1 -- start start 2024-02-12T16:30:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f75b91d6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75b40f6510 0x7f75b41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f75b91d6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b4106cd0 con 0x7f75b40f6510 2024-02-12T16:30:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f75b2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75b40f6510 0x7f75b41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f75b2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75b40f6510 0x7f75b41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58036/0 (socket says 172.21.15.104:58036) 2024-02-12T16:30:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f75b2d9d700 1 -- 172.21.15.104:0/3237296617 learned_addr learned my addr 172.21.15.104:0/3237296617 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-12T16:30:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f75b2d9d700 1 -- 172.21.15.104:0/3237296617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a8005040 con 0x7f75b40f6510 2024-02-12T16:30:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f75b2d9d700 1 --2- 172.21.15.104:0/3237296617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75b40f6510 0x7f75b41067e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f75a8005010 tx=0x7f75a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f759b7fe700 1 -- 172.21.15.104:0/3237296617 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a8009320 con 0x7f75b40f6510 2024-02-12T16:30:07.652 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f75b91d6700 1 -- 172.21.15.104:0/3237296617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b4106ed0 con 0x7f75b40f6510 2024-02-12T16:30:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f759b7fe700 1 -- 172.21.15.104:0/3237296617 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75a8013380 con 0x7f75b40f6510 2024-02-12T16:30:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f759b7fe700 1 -- 172.21.15.104:0/3237296617 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a8014e50 con 0x7f75b40f6510 2024-02-12T16:30:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.614+0000 7f75b91d6700 1 -- 172.21.15.104:0/3237296617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b4107360 con 0x7f75b40f6510 2024-02-12T16:30:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.615+0000 7f759b7fe700 1 -- 172.21.15.104:0/3237296617 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75a80263a0 con 0x7f75b40f6510 2024-02-12T16:30:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.615+0000 7f75b91d6700 1 -- 172.21.15.104:0/3237296617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b4045ad0 con 0x7f75b40f6510 2024-02-12T16:30:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.615+0000 7f759b7fe700 1 --2- 172.21.15.104:0/3237296617 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f759c02ff10 0x7f759c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.615+0000 7f759b7fe700 1 -- 172.21.15.104:0/3237296617 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75a8041150 con 0x7f75b40f6510 2024-02-12T16:30:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.616+0000 7f75b259c700 1 --2- 172.21.15.104:0/3237296617 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f759c02ff10 0x7f759c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.616+0000 7f75b259c700 1 --2- 172.21.15.104:0/3237296617 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f759c02ff10 0x7f759c0323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f75a40096a0 tx=0x7f75a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:07.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.620+0000 7f759b7fe700 1 -- 172.21.15.104:0/3237296617 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75a801ccf0 con 0x7f75b40f6510 2024-02-12T16:30:07.776 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:07.775+0000 7f75b91d6700 1 -- 172.21.15.104:0/3237296617 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f75b4108210 con 0x7f75b40f6510 2024-02-12T16:30:08.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:08.061+0000 7f759b7fe700 1 -- 172.21.15.104:0/3237296617 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f75a801cb10 con 0x7f75b40f6510 2024-02-12T16:30:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:08.064+0000 7f75b91d6700 1 -- 172.21.15.104:0/3237296617 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f759c02ff10 msgr2=0x7f759c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:08.064+0000 7f75b91d6700 1 --2- 172.21.15.104:0/3237296617 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f759c02ff10 0x7f759c0323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f75a40096a0 tx=0x7f75a4006b40).stop 2024-02-12T16:30:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:08.064+0000 7f75b91d6700 1 -- 172.21.15.104:0/3237296617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75b40f6510 msgr2=0x7f75b41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:08.064+0000 7f75b91d6700 1 --2- 172.21.15.104:0/3237296617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75b40f6510 0x7f75b41067e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f75a8005010 tx=0x7f75a800b040).stop 2024-02-12T16:30:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:08.064+0000 7f75b91d6700 1 -- 172.21.15.104:0/3237296617 shutdown_connections 2024-02-12T16:30:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:08.064+0000 7f75b91d6700 1 --2- 172.21.15.104:0/3237296617 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f759c02ff10 0x7f759c0323d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:08.064+0000 7f75b91d6700 1 --2- 172.21.15.104:0/3237296617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75b40f6510 0x7f75b41067e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:08.065 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:08.064+0000 7f75b91d6700 1 -- 172.21.15.104:0/3237296617 >> 172.21.15.104:0/3237296617 conn(0x7f75b40f1c70 msgr2=0x7f75b40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:08.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:08.064+0000 7f75b91d6700 1 -- 172.21.15.104:0/3237296617 shutdown_connections 2024-02-12T16:30:08.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-12T16:30:08.064+0000 7f75b91d6700 1 -- 172.21.15.104:0/3237296617 wait complete. 2024-02-12T16:30:08.066 INFO:teuthology.orchestra.run.smithi104.stderr:adjusted tunables profile to default 2024-02-12T16:30:08.395 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:06.278698+0000 mgr.smithi104.ohwbhf (mgr.14162) 20 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:08.395 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:07.775658+0000 mon.smithi104 (mon.0) 117 : audit [INF] from='client.? 172.21.15.104:0/3237296617' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-12T16:30:08.714 INFO:tasks.cephadm:Adding mon.smithi104 on smithi104 2024-02-12T16:30:08.714 INFO:tasks.cephadm:Adding mon.smithi149 on smithi149 2024-02-12T16:30:08.714 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph orch apply mon '2;smithi104:172.21.15.104=smithi104;smithi149:172.21.15.149=smithi149' 2024-02-12T16:30:09.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:08.061680+0000 mon.smithi104 (mon.0) 118 : audit [INF] from='client.? 172.21.15.104:0/3237296617' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-12T16:30:09.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:08.061752+0000 mon.smithi104 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-12T16:30:10.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:08.279021+0000 mgr.smithi104.ohwbhf (mgr.14162) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:10.442 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.430+0000 7f21c07a8700 1 -- 172.21.15.149:0/88740933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b80f5410 msgr2=0x7f21b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:10.442 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.430+0000 7f21c07a8700 1 --2- 172.21.15.149:0/88740933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b80f5410 0x7f21b80f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f21a8009fb0 tx=0x7f21a8009b30).stop 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.430+0000 7f21c07a8700 1 -- 172.21.15.149:0/88740933 shutdown_connections 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.430+0000 7f21c07a8700 1 --2- 172.21.15.149:0/88740933 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b80f5410 0x7f21b80f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.430+0000 7f21c07a8700 1 -- 172.21.15.149:0/88740933 >> 172.21.15.149:0/88740933 conn(0x7f21b80f0b70 msgr2=0x7f21b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.430+0000 7f21c07a8700 1 -- 172.21.15.149:0/88740933 shutdown_connections 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.430+0000 7f21c07a8700 1 -- 172.21.15.149:0/88740933 wait complete. 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.431+0000 7f21c07a8700 1 Processor -- start 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.431+0000 7f21c07a8700 1 -- start start 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.431+0000 7f21c07a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b80f5410 0x7f21b81094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.431+0000 7f21c07a8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b8060240 con 0x7f21b80f5410 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.431+0000 7f21be544700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b80f5410 0x7f21b81094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.431+0000 7f21be544700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b80f5410 0x7f21b81094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:37316/0 (socket says 172.21.15.149:37316) 2024-02-12T16:30:10.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.431+0000 7f21be544700 1 -- 172.21.15.149:0/1125110529 learned_addr learned my addr 172.21.15.149:0/1125110529 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.431+0000 7f21be544700 1 -- 172.21.15.149:0/1125110529 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a8005040 con 0x7f21b80f5410 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.432+0000 7f21be544700 1 --2- 172.21.15.149:0/1125110529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b80f5410 0x7f21b81094e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f21a80089f0 tx=0x7f21a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.432+0000 7f21aeffd700 1 -- 172.21.15.149:0/1125110529 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21a8009320 con 0x7f21b80f5410 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.432+0000 7f21c07a8700 1 -- 172.21.15.149:0/1125110529 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b8060440 con 0x7f21b80f5410 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.432+0000 7f21aeffd700 1 -- 172.21.15.149:0/1125110529 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21a8013380 con 0x7f21b80f5410 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.432+0000 7f21aeffd700 1 -- 172.21.15.149:0/1125110529 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21a801ed80 con 0x7f21b80f5410 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.432+0000 7f21c07a8700 1 -- 172.21.15.149:0/1125110529 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b8109b90 con 0x7f21b80f5410 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.433+0000 7f21aeffd700 1 -- 172.21.15.149:0/1125110529 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21a801e420 con 0x7f21b80f5410 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.433+0000 7f21c07a8700 1 -- 172.21.15.149:0/1125110529 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b8045ad0 con 0x7f21b80f5410 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.433+0000 7f21aeffd700 1 --2- 172.21.15.149:0/1125110529 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21a402ff60 0x7f21a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.433+0000 7f21aeffd700 1 -- 172.21.15.149:0/1125110529 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21a8041000 con 0x7f21b80f5410 2024-02-12T16:30:10.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.433+0000 7f21bdd43700 1 --2- 172.21.15.149:0/1125110529 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21a402ff60 0x7f21a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:10.445 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.434+0000 7f21bdd43700 1 --2- 172.21.15.149:0/1125110529 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21a402ff60 0x7f21a4032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f21b40096a0 tx=0x7f21b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:10.445 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.438+0000 7f21aeffd700 1 -- 172.21.15.149:0/1125110529 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21a80095f0 con 0x7f21b80f5410 2024-02-12T16:30:10.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.614+0000 7f21c07a8700 1 -- 172.21.15.149:0/1125110529 --> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi104:172.21.15.104=smithi104;smithi149:172.21.15.149=smithi149", "target": ["mon-mgr", ""]}) v1 -- 0x7f21b80f1dd0 con 0x7f21a402ff60 2024-02-12T16:30:10.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.620+0000 7f21aeffd700 1 -- 172.21.15.149:0/1125110529 <== mgr.14162 v2:172.21.15.104:6800/4169661527 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f21b80f1dd0 con 0x7f21a402ff60 2024-02-12T16:30:10.621 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled mon update... 2024-02-12T16:30:10.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.621+0000 7f21c07a8700 1 -- 172.21.15.149:0/1125110529 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21a402ff60 msgr2=0x7f21a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:10.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.621+0000 7f21c07a8700 1 --2- 172.21.15.149:0/1125110529 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21a402ff60 0x7f21a4032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f21b40096a0 tx=0x7f21b4006b40).stop 2024-02-12T16:30:10.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.621+0000 7f21c07a8700 1 -- 172.21.15.149:0/1125110529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b80f5410 msgr2=0x7f21b81094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:10.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.621+0000 7f21c07a8700 1 --2- 172.21.15.149:0/1125110529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b80f5410 0x7f21b81094e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f21a80089f0 tx=0x7f21a800b040).stop 2024-02-12T16:30:10.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.621+0000 7f21c07a8700 1 -- 172.21.15.149:0/1125110529 shutdown_connections 2024-02-12T16:30:10.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.621+0000 7f21c07a8700 1 --2- 172.21.15.149:0/1125110529 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21a402ff60 0x7f21a4032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:10.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.621+0000 7f21c07a8700 1 --2- 172.21.15.149:0/1125110529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21b80f5410 0x7f21b81094e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:10.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.621+0000 7f21c07a8700 1 -- 172.21.15.149:0/1125110529 >> 172.21.15.149:0/1125110529 conn(0x7f21b80f0b70 msgr2=0x7f21b80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:10.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.621+0000 7f21c07a8700 1 -- 172.21.15.149:0/1125110529 shutdown_connections 2024-02-12T16:30:10.624 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:10.621+0000 7f21c07a8700 1 -- 172.21.15.149:0/1125110529 wait complete. 2024-02-12T16:30:11.262 DEBUG:teuthology.orchestra.run.smithi149:mon.smithi149> sudo journalctl -f -n 0 -u ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi149.service 2024-02-12T16:30:11.265 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:11.265 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:30:11.293 INFO:journalctl@ceph.mon.smithi149.smithi149.stdout:-- Logs begin at Mon 2024-02-12 16:18:15 UTC. -- 2024-02-12T16:30:11.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:10.279349+0000 mgr.smithi104.ohwbhf (mgr.14162) 22 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:11.926 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:10.615654+0000 mgr.smithi104.ohwbhf (mgr.14162) 23 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi104:172.21.15.104=smithi104;smithi149:172.21.15.149=smithi149", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:30:11.926 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:30:10.617049+0000 mgr.smithi104.ohwbhf (mgr.14162) 24 : cephadm [INF] Saving service mon spec with placement smithi104:172.21.15.104=smithi104;smithi149:172.21.15.149=smithi149;count:2 2024-02-12T16:30:11.926 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:10.619678+0000 mon.smithi104 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:30:13.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.997+0000 7f5a5305b700 1 -- 172.21.15.149:0/540434286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4c0f51a0 msgr2=0x7f5a4c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:13.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.997+0000 7f5a5305b700 1 --2- 172.21.15.149:0/540434286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4c0f51a0 0x7f5a4c0f55c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5a48009fb0 tx=0x7f5a48009b30).stop 2024-02-12T16:30:13.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.997+0000 7f5a5305b700 1 -- 172.21.15.149:0/540434286 shutdown_connections 2024-02-12T16:30:13.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.997+0000 7f5a5305b700 1 --2- 172.21.15.149:0/540434286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4c0f51a0 0x7f5a4c0f55c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:13.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.997+0000 7f5a5305b700 1 -- 172.21.15.149:0/540434286 >> 172.21.15.149:0/540434286 conn(0x7f5a4c0f09a0 msgr2=0x7f5a4c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:13.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.997+0000 7f5a5305b700 1 -- 172.21.15.149:0/540434286 shutdown_connections 2024-02-12T16:30:13.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.997+0000 7f5a5305b700 1 -- 172.21.15.149:0/540434286 wait complete. 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.997+0000 7f5a5305b700 1 Processor -- start 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.997+0000 7f5a5305b700 1 -- start start 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.998+0000 7f5a5305b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4c0f51a0 0x7f5a4c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.998+0000 7f5a5305b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c1059c0 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.998+0000 7f5a52059700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4c0f51a0 0x7f5a4c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.998+0000 7f5a52059700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4c0f51a0 0x7f5a4c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:37336/0 (socket says 172.21.15.149:37336) 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.998+0000 7f5a52059700 1 -- 172.21.15.149:0/2667646096 learned_addr learned my addr 172.21.15.149:0/2667646096 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.998+0000 7f5a52059700 1 -- 172.21.15.149:0/2667646096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a48005040 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.998+0000 7f5a52059700 1 --2- 172.21.15.149:0/2667646096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4c0f51a0 0x7f5a4c1054d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f5a480089f0 tx=0x7f5a4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.998+0000 7f5a42ffd700 1 -- 172.21.15.149:0/2667646096 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a48009320 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.999+0000 7f5a5305b700 1 -- 172.21.15.149:0/2667646096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4c105bc0 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.999+0000 7f5a42ffd700 1 -- 172.21.15.149:0/2667646096 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a48013380 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.999+0000 7f5a42ffd700 1 -- 172.21.15.149:0/2667646096 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a48014e20 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.999+0000 7f5a5305b700 1 -- 172.21.15.149:0/2667646096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4c106050 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.999+0000 7f5a42ffd700 1 -- 172.21.15.149:0/2667646096 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a48014420 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.999+0000 7f5a5305b700 1 -- 172.21.15.149:0/2667646096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4c045ad0 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:12.999+0000 7f5a42ffd700 1 --2- 172.21.15.149:0/2667646096 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a3802ff10 0x7f5a380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:13.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.000+0000 7f5a42ffd700 1 -- 172.21.15.149:0/2667646096 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a48040f30 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.000+0000 7f5a51858700 1 --2- 172.21.15.149:0/2667646096 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a3802ff10 0x7f5a380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:13.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.000+0000 7f5a51858700 1 --2- 172.21.15.149:0/2667646096 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a3802ff10 0x7f5a380323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5a3c0096a0 tx=0x7f5a3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:13.006 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.004+0000 7f5a42ffd700 1 -- 172.21.15.149:0/2667646096 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a4801d980 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.210 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.208+0000 7f5a5305b700 1 -- 172.21.15.149:0/2667646096 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a4c106f00 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.211 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.209+0000 7f5a42ffd700 1 -- 172.21.15.149:0/2667646096 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5a4801d380 con 0x7f5a4c0f51a0 2024-02-12T16:30:13.211 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:13.211 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:30:13.213 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.210+0000 7f5a5305b700 1 -- 172.21.15.149:0/2667646096 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a3802ff10 msgr2=0x7f5a380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:13.213 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.210+0000 7f5a5305b700 1 --2- 172.21.15.149:0/2667646096 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a3802ff10 0x7f5a380323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5a3c0096a0 tx=0x7f5a3c006b40).stop 2024-02-12T16:30:13.213 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.210+0000 7f5a5305b700 1 -- 172.21.15.149:0/2667646096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4c0f51a0 msgr2=0x7f5a4c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:13.213 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.210+0000 7f5a5305b700 1 --2- 172.21.15.149:0/2667646096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4c0f51a0 0x7f5a4c1054d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f5a480089f0 tx=0x7f5a4800b040).stop 2024-02-12T16:30:13.213 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.210+0000 7f5a5305b700 1 -- 172.21.15.149:0/2667646096 shutdown_connections 2024-02-12T16:30:13.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.210+0000 7f5a5305b700 1 --2- 172.21.15.149:0/2667646096 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a3802ff10 0x7f5a380323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:13.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.210+0000 7f5a5305b700 1 --2- 172.21.15.149:0/2667646096 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a4c0f51a0 0x7f5a4c1054d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:13.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.210+0000 7f5a5305b700 1 -- 172.21.15.149:0/2667646096 >> 172.21.15.149:0/2667646096 conn(0x7f5a4c0f09a0 msgr2=0x7f5a4c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:13.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.210+0000 7f5a5305b700 1 -- 172.21.15.149:0/2667646096 shutdown_connections 2024-02-12T16:30:13.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:13.210+0000 7f5a5305b700 1 -- 172.21.15.149:0/2667646096 wait complete. 2024-02-12T16:30:13.214 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:30:14.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:12.279591+0000 mgr.smithi104.ohwbhf (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:14.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:13.210160+0000 mon.smithi104 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.149:0/2667646096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:30:14.811 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:14.812 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:30:16.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:14.279859+0000 mgr.smithi104.ohwbhf (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:16.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.469+0000 7f40e1b62700 1 -- 172.21.15.149:0/3353083288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc060280 msgr2=0x7f40dc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:16.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.469+0000 7f40e1b62700 1 --2- 172.21.15.149:0/3353083288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc060280 0x7f40dc0f90f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f40c4009fb0 tx=0x7f40c4009b30).stop 2024-02-12T16:30:16.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.469+0000 7f40e1b62700 1 -- 172.21.15.149:0/3353083288 shutdown_connections 2024-02-12T16:30:16.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.469+0000 7f40e1b62700 1 --2- 172.21.15.149:0/3353083288 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc060280 0x7f40dc0f90f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.469+0000 7f40e1b62700 1 -- 172.21.15.149:0/3353083288 >> 172.21.15.149:0/3353083288 conn(0x7f40dc0f0b70 msgr2=0x7f40dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.469+0000 7f40e1b62700 1 -- 172.21.15.149:0/3353083288 shutdown_connections 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.469+0000 7f40e1b62700 1 -- 172.21.15.149:0/3353083288 wait complete. 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.470+0000 7f40e1b62700 1 Processor -- start 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.470+0000 7f40e1b62700 1 -- start start 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.470+0000 7f40e1b62700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc060280 0x7f40dc106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.470+0000 7f40e1b62700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc106c70 con 0x7f40dc060280 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.470+0000 7f40db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc060280 0x7f40dc106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.470+0000 7f40db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc060280 0x7f40dc106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:32910/0 (socket says 172.21.15.149:32910) 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.470+0000 7f40db7fe700 1 -- 172.21.15.149:0/4003688732 learned_addr learned my addr 172.21.15.149:0/4003688732 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.471+0000 7f40db7fe700 1 -- 172.21.15.149:0/4003688732 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c4005040 con 0x7f40dc060280 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.471+0000 7f40db7fe700 1 --2- 172.21.15.149:0/4003688732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc060280 0x7f40dc106780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f40c40089f0 tx=0x7f40c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:16.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.471+0000 7f40d3fff700 1 -- 172.21.15.149:0/4003688732 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c4009320 con 0x7f40dc060280 2024-02-12T16:30:16.475 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.471+0000 7f40e1b62700 1 -- 172.21.15.149:0/4003688732 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc106e70 con 0x7f40dc060280 2024-02-12T16:30:16.475 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.471+0000 7f40d3fff700 1 -- 172.21.15.149:0/4003688732 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40c4013380 con 0x7f40dc060280 2024-02-12T16:30:16.475 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.471+0000 7f40d3fff700 1 -- 172.21.15.149:0/4003688732 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c4014e20 con 0x7f40dc060280 2024-02-12T16:30:16.475 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.471+0000 7f40e1b62700 1 -- 172.21.15.149:0/4003688732 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc107300 con 0x7f40dc060280 2024-02-12T16:30:16.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.472+0000 7f40d3fff700 1 -- 172.21.15.149:0/4003688732 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40c4014420 con 0x7f40dc060280 2024-02-12T16:30:16.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.472+0000 7f40d3fff700 1 --2- 172.21.15.149:0/4003688732 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f40c802ff10 0x7f40c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:16.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.472+0000 7f40e1b62700 1 -- 172.21.15.149:0/4003688732 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc045ad0 con 0x7f40dc060280 2024-02-12T16:30:16.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.472+0000 7f40d3fff700 1 -- 172.21.15.149:0/4003688732 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40c4040f50 con 0x7f40dc060280 2024-02-12T16:30:16.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.472+0000 7f40daffd700 1 --2- 172.21.15.149:0/4003688732 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f40c802ff10 0x7f40c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:16.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.473+0000 7f40daffd700 1 --2- 172.21.15.149:0/4003688732 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f40c802ff10 0x7f40c80323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f40cc0096a0 tx=0x7f40cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:16.479 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.477+0000 7f40d3fff700 1 -- 172.21.15.149:0/4003688732 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40c40095f0 con 0x7f40dc060280 2024-02-12T16:30:16.685 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.683+0000 7f40e1b62700 1 -- 172.21.15.149:0/4003688732 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40dc1081b0 con 0x7f40dc060280 2024-02-12T16:30:16.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.683+0000 7f40d3fff700 1 -- 172.21.15.149:0/4003688732 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f40c401db70 con 0x7f40dc060280 2024-02-12T16:30:16.686 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:16.686 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:30:16.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.685+0000 7f40e1b62700 1 -- 172.21.15.149:0/4003688732 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f40c802ff10 msgr2=0x7f40c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:16.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.685+0000 7f40e1b62700 1 --2- 172.21.15.149:0/4003688732 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f40c802ff10 0x7f40c80323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f40cc0096a0 tx=0x7f40cc006b40).stop 2024-02-12T16:30:16.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.685+0000 7f40e1b62700 1 -- 172.21.15.149:0/4003688732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc060280 msgr2=0x7f40dc106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:16.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.685+0000 7f40e1b62700 1 --2- 172.21.15.149:0/4003688732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc060280 0x7f40dc106780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f40c40089f0 tx=0x7f40c400b040).stop 2024-02-12T16:30:16.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.685+0000 7f40e1b62700 1 -- 172.21.15.149:0/4003688732 shutdown_connections 2024-02-12T16:30:16.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.685+0000 7f40e1b62700 1 --2- 172.21.15.149:0/4003688732 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f40c802ff10 0x7f40c80323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:16.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.685+0000 7f40e1b62700 1 --2- 172.21.15.149:0/4003688732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40dc060280 0x7f40dc106780 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:16.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.685+0000 7f40e1b62700 1 -- 172.21.15.149:0/4003688732 >> 172.21.15.149:0/4003688732 conn(0x7f40dc0f0b70 msgr2=0x7f40dc0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:16.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.685+0000 7f40e1b62700 1 -- 172.21.15.149:0/4003688732 shutdown_connections 2024-02-12T16:30:16.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:16.685+0000 7f40e1b62700 1 -- 172.21.15.149:0/4003688732 wait complete. 2024-02-12T16:30:16.689 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:30:17.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:17 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:16.684805+0000 mon.smithi104 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.149:0/4003688732' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:30:18.351 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:18.352 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:30:18.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:16.280223+0000 mgr.smithi104.ohwbhf (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:20.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:18.280589+0000 mgr.smithi104.ohwbhf (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:20.522 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.519+0000 7f916b11a700 1 -- 172.21.15.149:0/3350591663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91640f4920 msgr2=0x7f91640f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:20.522 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.519+0000 7f916b11a700 1 --2- 172.21.15.149:0/3350591663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91640f4920 0x7f91640f4d40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f9160009fb0 tx=0x7f9160009b30).stop 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.519+0000 7f916b11a700 1 -- 172.21.15.149:0/3350591663 shutdown_connections 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.519+0000 7f916b11a700 1 --2- 172.21.15.149:0/3350591663 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91640f4920 0x7f91640f4d40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.519+0000 7f916b11a700 1 -- 172.21.15.149:0/3350591663 >> 172.21.15.149:0/3350591663 conn(0x7f91640f0120 msgr2=0x7f91640f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.519+0000 7f916b11a700 1 -- 172.21.15.149:0/3350591663 shutdown_connections 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.519+0000 7f916b11a700 1 -- 172.21.15.149:0/3350591663 wait complete. 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.519+0000 7f916b11a700 1 Processor -- start 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.519+0000 7f916b11a700 1 -- start start 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.519+0000 7f916b11a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91640f4920 0x7f91641092a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.519+0000 7f916b11a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91640f1670 con 0x7f91640f4920 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.520+0000 7f916a118700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91640f4920 0x7f91641092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.520+0000 7f916a118700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91640f4920 0x7f91641092a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:32922/0 (socket says 172.21.15.149:32922) 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.520+0000 7f916a118700 1 -- 172.21.15.149:0/1093185080 learned_addr learned my addr 172.21.15.149:0/1093185080 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:20.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.520+0000 7f916a118700 1 -- 172.21.15.149:0/1093185080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9160005040 con 0x7f91640f4920 2024-02-12T16:30:20.524 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.520+0000 7f916a118700 1 --2- 172.21.15.149:0/1093185080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91640f4920 0x7f91641092a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f91600089f0 tx=0x7f916000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:20.524 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.520+0000 7f915affd700 1 -- 172.21.15.149:0/1093185080 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9160009320 con 0x7f91640f4920 2024-02-12T16:30:20.524 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.520+0000 7f916b11a700 1 -- 172.21.15.149:0/1093185080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91640f1870 con 0x7f91640f4920 2024-02-12T16:30:20.524 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.520+0000 7f915affd700 1 -- 172.21.15.149:0/1093185080 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9160013380 con 0x7f91640f4920 2024-02-12T16:30:20.524 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.520+0000 7f915affd700 1 -- 172.21.15.149:0/1093185080 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9160014e20 con 0x7f91640f4920 2024-02-12T16:30:20.524 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.520+0000 7f916b11a700 1 -- 172.21.15.149:0/1093185080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9164109950 con 0x7f91640f4920 2024-02-12T16:30:20.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.521+0000 7f915affd700 1 -- 172.21.15.149:0/1093185080 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9160014420 con 0x7f91640f4920 2024-02-12T16:30:20.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.521+0000 7f916b11a700 1 -- 172.21.15.149:0/1093185080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9164045ad0 con 0x7f91640f4920 2024-02-12T16:30:20.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.521+0000 7f915affd700 1 --2- 172.21.15.149:0/1093185080 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f915002ff10 0x7f91500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:20.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.521+0000 7f915affd700 1 -- 172.21.15.149:0/1093185080 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9160040e70 con 0x7f91640f4920 2024-02-12T16:30:20.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.521+0000 7f9169917700 1 --2- 172.21.15.149:0/1093185080 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f915002ff10 0x7f91500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:20.525 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.522+0000 7f9169917700 1 --2- 172.21.15.149:0/1093185080 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f915002ff10 0x7f91500323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f91540096a0 tx=0x7f9154006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:20.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.526+0000 7f915affd700 1 -- 172.21.15.149:0/1093185080 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f916001d980 con 0x7f91640f4920 2024-02-12T16:30:20.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.734+0000 7f916b11a700 1 -- 172.21.15.149:0/1093185080 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9164027a00 con 0x7f91640f4920 2024-02-12T16:30:20.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.735+0000 7f915affd700 1 -- 172.21.15.149:0/1093185080 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f916001d380 con 0x7f91640f4920 2024-02-12T16:30:20.737 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:20.738 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:30:20.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.737+0000 7f916b11a700 1 -- 172.21.15.149:0/1093185080 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f915002ff10 msgr2=0x7f91500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:20.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.737+0000 7f916b11a700 1 --2- 172.21.15.149:0/1093185080 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f915002ff10 0x7f91500323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f91540096a0 tx=0x7f9154006b40).stop 2024-02-12T16:30:20.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.737+0000 7f916b11a700 1 -- 172.21.15.149:0/1093185080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91640f4920 msgr2=0x7f91641092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:20.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.737+0000 7f916b11a700 1 --2- 172.21.15.149:0/1093185080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91640f4920 0x7f91641092a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f91600089f0 tx=0x7f916000b040).stop 2024-02-12T16:30:20.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.737+0000 7f916b11a700 1 -- 172.21.15.149:0/1093185080 shutdown_connections 2024-02-12T16:30:20.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.737+0000 7f916b11a700 1 --2- 172.21.15.149:0/1093185080 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f915002ff10 0x7f91500323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:20.740 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.737+0000 7f916b11a700 1 --2- 172.21.15.149:0/1093185080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f91640f4920 0x7f91641092a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:20.740 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.737+0000 7f916b11a700 1 -- 172.21.15.149:0/1093185080 >> 172.21.15.149:0/1093185080 conn(0x7f91640f0120 msgr2=0x7f91640f22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:20.740 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.737+0000 7f916b11a700 1 -- 172.21.15.149:0/1093185080 shutdown_connections 2024-02-12T16:30:20.740 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:20.737+0000 7f916b11a700 1 -- 172.21.15.149:0/1093185080 wait complete. 2024-02-12T16:30:20.740 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:30:21.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:21 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:20.736444+0000 mon.smithi104 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.149:0/1093185080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:30:22.317 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:22.318 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:30:22.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:20.280870+0000 mgr.smithi104.ohwbhf (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:24.031 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.028+0000 7fdcbeee7700 1 -- 172.21.15.149:0/620179093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcb80f51a0 msgr2=0x7fdcb80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:24.031 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.028+0000 7fdcbeee7700 1 --2- 172.21.15.149:0/620179093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcb80f51a0 0x7fdcb80f55c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdcb4009fb0 tx=0x7fdcb4009b30).stop 2024-02-12T16:30:24.031 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.028+0000 7fdcbeee7700 1 -- 172.21.15.149:0/620179093 shutdown_connections 2024-02-12T16:30:24.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.028+0000 7fdcbeee7700 1 --2- 172.21.15.149:0/620179093 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcb80f51a0 0x7fdcb80f55c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:24.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.028+0000 7fdcbeee7700 1 -- 172.21.15.149:0/620179093 >> 172.21.15.149:0/620179093 conn(0x7fdcb80f09a0 msgr2=0x7fdcb80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:24.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.028+0000 7fdcbeee7700 1 -- 172.21.15.149:0/620179093 shutdown_connections 2024-02-12T16:30:24.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.028+0000 7fdcbeee7700 1 -- 172.21.15.149:0/620179093 wait complete. 2024-02-12T16:30:24.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.029+0000 7fdcbeee7700 1 Processor -- start 2024-02-12T16:30:24.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.029+0000 7fdcbeee7700 1 -- start start 2024-02-12T16:30:24.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.029+0000 7fdcbeee7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcb80f51a0 0x7fdcb81054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:24.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.029+0000 7fdcbeee7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb81059c0 con 0x7fdcb80f51a0 2024-02-12T16:30:24.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.029+0000 7fdcbdee5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcb80f51a0 0x7fdcb81054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:24.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.029+0000 7fdcbdee5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcb80f51a0 0x7fdcb81054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:32936/0 (socket says 172.21.15.149:32936) 2024-02-12T16:30:24.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.029+0000 7fdcbdee5700 1 -- 172.21.15.149:0/283823045 learned_addr learned my addr 172.21.15.149:0/283823045 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:24.034 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.030+0000 7fdcbdee5700 1 -- 172.21.15.149:0/283823045 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb4005040 con 0x7fdcb80f51a0 2024-02-12T16:30:24.034 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.030+0000 7fdcbdee5700 1 --2- 172.21.15.149:0/283823045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcb80f51a0 0x7fdcb81054d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fdcb40089f0 tx=0x7fdcb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:24.034 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.030+0000 7fdcae7fc700 1 -- 172.21.15.149:0/283823045 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcb4009320 con 0x7fdcb80f51a0 2024-02-12T16:30:24.034 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.030+0000 7fdcbeee7700 1 -- 172.21.15.149:0/283823045 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb8105bc0 con 0x7fdcb80f51a0 2024-02-12T16:30:24.034 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.030+0000 7fdcae7fc700 1 -- 172.21.15.149:0/283823045 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcb4013380 con 0x7fdcb80f51a0 2024-02-12T16:30:24.034 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.030+0000 7fdcae7fc700 1 -- 172.21.15.149:0/283823045 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcb4014e20 con 0x7fdcb80f51a0 2024-02-12T16:30:24.034 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.030+0000 7fdcbeee7700 1 -- 172.21.15.149:0/283823045 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb8106050 con 0x7fdcb80f51a0 2024-02-12T16:30:24.034 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.031+0000 7fdcae7fc700 1 -- 172.21.15.149:0/283823045 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcb4014420 con 0x7fdcb80f51a0 2024-02-12T16:30:24.035 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.031+0000 7fdcbeee7700 1 -- 172.21.15.149:0/283823045 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb8045ad0 con 0x7fdcb80f51a0 2024-02-12T16:30:24.035 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.031+0000 7fdcae7fc700 1 --2- 172.21.15.149:0/283823045 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdca402ff10 0x7fdca40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:24.035 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.031+0000 7fdcae7fc700 1 -- 172.21.15.149:0/283823045 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcb4040fc0 con 0x7fdcb80f51a0 2024-02-12T16:30:24.035 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.031+0000 7fdcbd6e4700 1 --2- 172.21.15.149:0/283823045 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdca402ff10 0x7fdca40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:24.035 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.032+0000 7fdcbd6e4700 1 --2- 172.21.15.149:0/283823045 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdca402ff10 0x7fdca40323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdca80096a0 tx=0x7fdca8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:24.038 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.036+0000 7fdcae7fc700 1 -- 172.21.15.149:0/283823045 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcb406d2b0 con 0x7fdcb80f51a0 2024-02-12T16:30:24.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.242+0000 7fdcbeee7700 1 -- 172.21.15.149:0/283823045 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcb8106f00 con 0x7fdcb80f51a0 2024-02-12T16:30:24.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.243+0000 7fdcae7fc700 1 -- 172.21.15.149:0/283823045 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdcb40095f0 con 0x7fdcb80f51a0 2024-02-12T16:30:24.245 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:24.245 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:30:24.246 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.244+0000 7fdcbeee7700 1 -- 172.21.15.149:0/283823045 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdca402ff10 msgr2=0x7fdca40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:24.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.244+0000 7fdcbeee7700 1 --2- 172.21.15.149:0/283823045 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdca402ff10 0x7fdca40323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdca80096a0 tx=0x7fdca8006b40).stop 2024-02-12T16:30:24.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.244+0000 7fdcbeee7700 1 -- 172.21.15.149:0/283823045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcb80f51a0 msgr2=0x7fdcb81054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:24.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.244+0000 7fdcbeee7700 1 --2- 172.21.15.149:0/283823045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcb80f51a0 0x7fdcb81054d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fdcb40089f0 tx=0x7fdcb400b040).stop 2024-02-12T16:30:24.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.244+0000 7fdcbeee7700 1 -- 172.21.15.149:0/283823045 shutdown_connections 2024-02-12T16:30:24.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.244+0000 7fdcbeee7700 1 --2- 172.21.15.149:0/283823045 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdca402ff10 0x7fdca40323d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:24.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.244+0000 7fdcbeee7700 1 --2- 172.21.15.149:0/283823045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcb80f51a0 0x7fdcb81054d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:24.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.244+0000 7fdcbeee7700 1 -- 172.21.15.149:0/283823045 >> 172.21.15.149:0/283823045 conn(0x7fdcb80f09a0 msgr2=0x7fdcb80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:24.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.245+0000 7fdcbeee7700 1 -- 172.21.15.149:0/283823045 shutdown_connections 2024-02-12T16:30:24.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:24.245+0000 7fdcbeee7700 1 -- 172.21.15.149:0/283823045 wait complete. 2024-02-12T16:30:24.248 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:30:24.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:22.281240+0000 mgr.smithi104.ohwbhf (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:24.243832+0000 mon.smithi104 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.149:0/283823045' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:30:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:24.402313+0000 mon.smithi104 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:30:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:24.409715+0000 mon.smithi104 (mon.0) 126 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:30:26.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:24.281554+0000 mgr.smithi104.ohwbhf (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:28.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:26.281912+0000 mgr.smithi104.ohwbhf (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:29.945 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:29.945 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:30:30.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:28.282264+0000 mgr.smithi104.ohwbhf (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:32.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:30.282530+0000 mgr.smithi104.ohwbhf (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:34.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:32.282855+0000 mgr.smithi104.ohwbhf (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:35.997 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.994+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/4112474498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b640f5410 msgr2=0x7f5b640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:35.998 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.994+0000 7f5b6a0ad700 1 --2- 172.21.15.149:0/4112474498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b640f5410 0x7f5b640f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f5b54009fb0 tx=0x7f5b54009b30).stop 2024-02-12T16:30:35.998 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.994+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/4112474498 shutdown_connections 2024-02-12T16:30:35.998 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.994+0000 7f5b6a0ad700 1 --2- 172.21.15.149:0/4112474498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b640f5410 0x7f5b640f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:35.998 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.994+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/4112474498 >> 172.21.15.149:0/4112474498 conn(0x7f5b640f0b70 msgr2=0x7f5b640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:35.998 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.994+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/4112474498 shutdown_connections 2024-02-12T16:30:35.998 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.995+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/4112474498 wait complete. 2024-02-12T16:30:35.998 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.995+0000 7f5b6a0ad700 1 Processor -- start 2024-02-12T16:30:35.998 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.995+0000 7f5b6a0ad700 1 -- start start 2024-02-12T16:30:35.998 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b6a0ad700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b640f5410 0x7f5b641056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:35.999 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b6a0ad700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b64105bd0 con 0x7f5b640f5410 2024-02-12T16:30:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b637fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b640f5410 0x7f5b641056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b637fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b640f5410 0x7f5b641056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33108/0 (socket says 172.21.15.149:33108) 2024-02-12T16:30:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b637fe700 1 -- 172.21.15.149:0/2175151108 learned_addr learned my addr 172.21.15.149:0/2175151108 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b637fe700 1 -- 172.21.15.149:0/2175151108 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b54005040 con 0x7f5b640f5410 2024-02-12T16:30:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b637fe700 1 --2- 172.21.15.149:0/2175151108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b640f5410 0x7f5b641056e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5b540089f0 tx=0x7f5b5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b5bfff700 1 -- 172.21.15.149:0/2175151108 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b54009320 con 0x7f5b640f5410 2024-02-12T16:30:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b5bfff700 1 -- 172.21.15.149:0/2175151108 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b54013380 con 0x7f5b640f5410 2024-02-12T16:30:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/2175151108 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b64105dd0 con 0x7f5b640f5410 2024-02-12T16:30:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b5bfff700 1 -- 172.21.15.149:0/2175151108 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b5401ed80 con 0x7f5b640f5410 2024-02-12T16:30:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.996+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/2175151108 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b64106260 con 0x7f5b640f5410 2024-02-12T16:30:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.997+0000 7f5b5bfff700 1 -- 172.21.15.149:0/2175151108 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b5401e420 con 0x7f5b640f5410 2024-02-12T16:30:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.997+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/2175151108 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b64045ad0 con 0x7f5b640f5410 2024-02-12T16:30:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.997+0000 7f5b5bfff700 1 --2- 172.21.15.149:0/2175151108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b5002ff60 0x7f5b50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.997+0000 7f5b5bfff700 1 -- 172.21.15.149:0/2175151108 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b54040ed0 con 0x7f5b640f5410 2024-02-12T16:30:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.998+0000 7f5b62ffd700 1 --2- 172.21.15.149:0/2175151108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b5002ff60 0x7f5b50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:35.998+0000 7f5b62ffd700 1 --2- 172.21.15.149:0/2175151108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b5002ff60 0x7f5b50032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f5b4c0096a0 tx=0x7f5b4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:36.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.002+0000 7f5b5bfff700 1 -- 172.21.15.149:0/2175151108 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b5401c930 con 0x7f5b640f5410 2024-02-12T16:30:36.208 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.205+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/2175151108 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b64107110 con 0x7f5b640f5410 2024-02-12T16:30:36.208 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.206+0000 7f5b5bfff700 1 -- 172.21.15.149:0/2175151108 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5b5401cb40 con 0x7f5b640f5410 2024-02-12T16:30:36.208 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:36.209 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:30:36.210 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.208+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/2175151108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b5002ff60 msgr2=0x7f5b50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:36.210 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.208+0000 7f5b6a0ad700 1 --2- 172.21.15.149:0/2175151108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b5002ff60 0x7f5b50032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f5b4c0096a0 tx=0x7f5b4c006b40).stop 2024-02-12T16:30:36.210 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.208+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/2175151108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b640f5410 msgr2=0x7f5b641056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:36.210 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.208+0000 7f5b6a0ad700 1 --2- 172.21.15.149:0/2175151108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b640f5410 0x7f5b641056e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5b540089f0 tx=0x7f5b5400b040).stop 2024-02-12T16:30:36.211 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.208+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/2175151108 shutdown_connections 2024-02-12T16:30:36.211 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.208+0000 7f5b6a0ad700 1 --2- 172.21.15.149:0/2175151108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b5002ff60 0x7f5b50032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:36.211 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.208+0000 7f5b6a0ad700 1 --2- 172.21.15.149:0/2175151108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b640f5410 0x7f5b641056e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:36.211 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.208+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/2175151108 >> 172.21.15.149:0/2175151108 conn(0x7f5b640f0b70 msgr2=0x7f5b640f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:36.211 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.208+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/2175151108 shutdown_connections 2024-02-12T16:30:36.211 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:36.208+0000 7f5b6a0ad700 1 -- 172.21.15.149:0/2175151108 wait complete. 2024-02-12T16:30:36.211 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:30:36.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:34.283208+0000 mgr.smithi104.ohwbhf (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:37.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:37 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:36.207410+0000 mon.smithi104 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.149:0/2175151108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:30:37.793 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:37.794 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:30:38.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:36.283447+0000 mgr.smithi104.ohwbhf (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:39.703 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.699+0000 7fb4716b6700 1 -- 172.21.15.149:0/1186201560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb46c0f6510 msgr2=0x7fb46c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:39.703 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.699+0000 7fb4716b6700 1 --2- 172.21.15.149:0/1186201560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb46c0f6510 0x7fb46c0f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fb45c009fb0 tx=0x7fb45c009b30).stop 2024-02-12T16:30:39.703 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.699+0000 7fb4716b6700 1 -- 172.21.15.149:0/1186201560 shutdown_connections 2024-02-12T16:30:39.703 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.699+0000 7fb4716b6700 1 --2- 172.21.15.149:0/1186201560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb46c0f6510 0x7fb46c0f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:39.703 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.699+0000 7fb4716b6700 1 -- 172.21.15.149:0/1186201560 >> 172.21.15.149:0/1186201560 conn(0x7fb46c0f1c70 msgr2=0x7fb46c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.699+0000 7fb4716b6700 1 -- 172.21.15.149:0/1186201560 shutdown_connections 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.699+0000 7fb4716b6700 1 -- 172.21.15.149:0/1186201560 wait complete. 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.700+0000 7fb4716b6700 1 Processor -- start 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.700+0000 7fb4716b6700 1 -- start start 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.700+0000 7fb4716b6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb46c0f6510 0x7fb46c1089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.700+0000 7fb4716b6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c108eb0 con 0x7fb46c0f6510 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.700+0000 7fb46affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb46c0f6510 0x7fb46c1089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.700+0000 7fb46affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb46c0f6510 0x7fb46c1089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33128/0 (socket says 172.21.15.149:33128) 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.700+0000 7fb46affd700 1 -- 172.21.15.149:0/2906783464 learned_addr learned my addr 172.21.15.149:0/2906783464 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.701+0000 7fb46affd700 1 -- 172.21.15.149:0/2906783464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c005040 con 0x7fb46c0f6510 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.701+0000 7fb46affd700 1 --2- 172.21.15.149:0/2906783464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb46c0f6510 0x7fb46c1089c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fb45c005010 tx=0x7fb45c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.701+0000 7fb4537fe700 1 -- 172.21.15.149:0/2906783464 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb45c009320 con 0x7fb46c0f6510 2024-02-12T16:30:39.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.701+0000 7fb4716b6700 1 -- 172.21.15.149:0/2906783464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46c1090b0 con 0x7fb46c0f6510 2024-02-12T16:30:39.705 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.701+0000 7fb4537fe700 1 -- 172.21.15.149:0/2906783464 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb45c013380 con 0x7fb46c0f6510 2024-02-12T16:30:39.705 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.701+0000 7fb4537fe700 1 -- 172.21.15.149:0/2906783464 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb45c014e50 con 0x7fb46c0f6510 2024-02-12T16:30:39.705 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.701+0000 7fb4716b6700 1 -- 172.21.15.149:0/2906783464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c109540 con 0x7fb46c0f6510 2024-02-12T16:30:39.705 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.702+0000 7fb4537fe700 1 -- 172.21.15.149:0/2906783464 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb45c014420 con 0x7fb46c0f6510 2024-02-12T16:30:39.706 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.702+0000 7fb4716b6700 1 -- 172.21.15.149:0/2906783464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c045ad0 con 0x7fb46c0f6510 2024-02-12T16:30:39.706 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.702+0000 7fb4537fe700 1 --2- 172.21.15.149:0/2906783464 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb45402ff10 0x7fb4540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:39.706 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.702+0000 7fb4537fe700 1 -- 172.21.15.149:0/2906783464 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb45c040e80 con 0x7fb46c0f6510 2024-02-12T16:30:39.706 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.702+0000 7fb46a7fc700 1 --2- 172.21.15.149:0/2906783464 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb45402ff10 0x7fb4540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:39.706 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.703+0000 7fb46a7fc700 1 --2- 172.21.15.149:0/2906783464 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb45402ff10 0x7fb4540323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fb4600096a0 tx=0x7fb460006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:39.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.707+0000 7fb4537fe700 1 -- 172.21.15.149:0/2906783464 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb45c014790 con 0x7fb46c0f6510 2024-02-12T16:30:39.912 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.910+0000 7fb4716b6700 1 -- 172.21.15.149:0/2906783464 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb46c10a3f0 con 0x7fb46c0f6510 2024-02-12T16:30:39.913 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.911+0000 7fb4537fe700 1 -- 172.21.15.149:0/2906783464 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb45c01d380 con 0x7fb46c0f6510 2024-02-12T16:30:39.913 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:39.913 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:30:39.915 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.912+0000 7fb4716b6700 1 -- 172.21.15.149:0/2906783464 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb45402ff10 msgr2=0x7fb4540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:39.915 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.912+0000 7fb4716b6700 1 --2- 172.21.15.149:0/2906783464 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb45402ff10 0x7fb4540323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fb4600096a0 tx=0x7fb460006b40).stop 2024-02-12T16:30:39.915 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.912+0000 7fb4716b6700 1 -- 172.21.15.149:0/2906783464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb46c0f6510 msgr2=0x7fb46c1089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:39.915 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.912+0000 7fb4716b6700 1 --2- 172.21.15.149:0/2906783464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb46c0f6510 0x7fb46c1089c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fb45c005010 tx=0x7fb45c00b040).stop 2024-02-12T16:30:39.915 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.912+0000 7fb4716b6700 1 -- 172.21.15.149:0/2906783464 shutdown_connections 2024-02-12T16:30:39.915 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.912+0000 7fb4716b6700 1 --2- 172.21.15.149:0/2906783464 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb45402ff10 0x7fb4540323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:39.915 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.912+0000 7fb4716b6700 1 --2- 172.21.15.149:0/2906783464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb46c0f6510 0x7fb46c1089c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:39.915 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.912+0000 7fb4716b6700 1 -- 172.21.15.149:0/2906783464 >> 172.21.15.149:0/2906783464 conn(0x7fb46c0f1c70 msgr2=0x7fb46c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:39.915 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.912+0000 7fb4716b6700 1 -- 172.21.15.149:0/2906783464 shutdown_connections 2024-02-12T16:30:39.915 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:39.912+0000 7fb4716b6700 1 -- 172.21.15.149:0/2906783464 wait complete. 2024-02-12T16:30:39.915 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:30:40.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:38.283719+0000 mgr.smithi104.ohwbhf (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:40.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:39.911941+0000 mon.smithi104 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.149:0/2906783464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:30:41.454 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:41.455 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:30:42.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:40.284077+0000 mgr.smithi104.ohwbhf (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:43.239 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.236+0000 7fc63359e700 1 -- 172.21.15.149:0/2366353390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6340f62a0 msgr2=0x7fc6340f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:43.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.236+0000 7fc63359e700 1 --2- 172.21.15.149:0/2366353390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6340f62a0 0x7fc6340f66c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fc624009fb0 tx=0x7fc624009b30).stop 2024-02-12T16:30:43.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.236+0000 7fc63359e700 1 -- 172.21.15.149:0/2366353390 shutdown_connections 2024-02-12T16:30:43.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.236+0000 7fc63359e700 1 --2- 172.21.15.149:0/2366353390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6340f62a0 0x7fc6340f66c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:43.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.236+0000 7fc63359e700 1 -- 172.21.15.149:0/2366353390 >> 172.21.15.149:0/2366353390 conn(0x7fc6340f1aa0 msgr2=0x7fc6340f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:43.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.236+0000 7fc63359e700 1 -- 172.21.15.149:0/2366353390 shutdown_connections 2024-02-12T16:30:43.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.236+0000 7fc63359e700 1 -- 172.21.15.149:0/2366353390 wait complete. 2024-02-12T16:30:43.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.236+0000 7fc63359e700 1 Processor -- start 2024-02-12T16:30:43.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.236+0000 7fc63359e700 1 -- start start 2024-02-12T16:30:43.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.236+0000 7fc63359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6340f62a0 0x7fc6341087e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:43.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.236+0000 7fc63359e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc634108cd0 con 0x7fc6340f62a0 2024-02-12T16:30:43.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.237+0000 7fc63259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6340f62a0 0x7fc6341087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:43.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.237+0000 7fc63259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6340f62a0 0x7fc6341087e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33150/0 (socket says 172.21.15.149:33150) 2024-02-12T16:30:43.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.237+0000 7fc63259c700 1 -- 172.21.15.149:0/2886603046 learned_addr learned my addr 172.21.15.149:0/2886603046 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:43.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.237+0000 7fc63259c700 1 -- 172.21.15.149:0/2886603046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc624005040 con 0x7fc6340f62a0 2024-02-12T16:30:43.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.237+0000 7fc63259c700 1 --2- 172.21.15.149:0/2886603046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6340f62a0 0x7fc6341087e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc6240089f0 tx=0x7fc62400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:43.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.237+0000 7fc62affd700 1 -- 172.21.15.149:0/2886603046 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc624009320 con 0x7fc6340f62a0 2024-02-12T16:30:43.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.237+0000 7fc62affd700 1 -- 172.21.15.149:0/2886603046 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc624013380 con 0x7fc6340f62a0 2024-02-12T16:30:43.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.237+0000 7fc63359e700 1 -- 172.21.15.149:0/2886603046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc634108ed0 con 0x7fc6340f62a0 2024-02-12T16:30:43.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.237+0000 7fc62affd700 1 -- 172.21.15.149:0/2886603046 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc624014e20 con 0x7fc6340f62a0 2024-02-12T16:30:43.242 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.237+0000 7fc63359e700 1 -- 172.21.15.149:0/2886603046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc634109360 con 0x7fc6340f62a0 2024-02-12T16:30:43.242 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.238+0000 7fc62affd700 1 -- 172.21.15.149:0/2886603046 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc624014420 con 0x7fc6340f62a0 2024-02-12T16:30:43.242 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.238+0000 7fc63359e700 1 -- 172.21.15.149:0/2886603046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc634045ad0 con 0x7fc6340f62a0 2024-02-12T16:30:43.242 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.238+0000 7fc62affd700 1 --2- 172.21.15.149:0/2886603046 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc62002ff10 0x7fc6200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:43.242 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.238+0000 7fc62affd700 1 -- 172.21.15.149:0/2886603046 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc624040f50 con 0x7fc6340f62a0 2024-02-12T16:30:43.242 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.239+0000 7fc631d9b700 1 --2- 172.21.15.149:0/2886603046 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc62002ff10 0x7fc6200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:43.242 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.239+0000 7fc631d9b700 1 --2- 172.21.15.149:0/2886603046 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc62002ff10 0x7fc6200323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc61c0096a0 tx=0x7fc61c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:43.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.243+0000 7fc62affd700 1 -- 172.21.15.149:0/2886603046 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc62406d2b0 con 0x7fc6340f62a0 2024-02-12T16:30:43.451 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.449+0000 7fc63359e700 1 -- 172.21.15.149:0/2886603046 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc63410a210 con 0x7fc6340f62a0 2024-02-12T16:30:43.451 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.449+0000 7fc62affd700 1 -- 172.21.15.149:0/2886603046 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc6240095f0 con 0x7fc6340f62a0 2024-02-12T16:30:43.452 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:43.452 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:30:43.453 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.451+0000 7fc63359e700 1 -- 172.21.15.149:0/2886603046 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc62002ff10 msgr2=0x7fc6200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:43.453 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.451+0000 7fc63359e700 1 --2- 172.21.15.149:0/2886603046 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc62002ff10 0x7fc6200323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc61c0096a0 tx=0x7fc61c006b40).stop 2024-02-12T16:30:43.453 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.451+0000 7fc63359e700 1 -- 172.21.15.149:0/2886603046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6340f62a0 msgr2=0x7fc6341087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:43.453 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.451+0000 7fc63359e700 1 --2- 172.21.15.149:0/2886603046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6340f62a0 0x7fc6341087e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc6240089f0 tx=0x7fc62400b040).stop 2024-02-12T16:30:43.454 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.451+0000 7fc63359e700 1 -- 172.21.15.149:0/2886603046 shutdown_connections 2024-02-12T16:30:43.454 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.451+0000 7fc63359e700 1 --2- 172.21.15.149:0/2886603046 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc62002ff10 0x7fc6200323d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:43.454 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.451+0000 7fc63359e700 1 --2- 172.21.15.149:0/2886603046 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6340f62a0 0x7fc6341087e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:43.454 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.451+0000 7fc63359e700 1 -- 172.21.15.149:0/2886603046 >> 172.21.15.149:0/2886603046 conn(0x7fc6340f1aa0 msgr2=0x7fc6340f2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:43.454 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.451+0000 7fc63359e700 1 -- 172.21.15.149:0/2886603046 shutdown_connections 2024-02-12T16:30:43.454 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:43.451+0000 7fc63359e700 1 -- 172.21.15.149:0/2886603046 wait complete. 2024-02-12T16:30:43.454 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:30:44.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:42.284470+0000 mgr.smithi104.ohwbhf (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:44.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:43.450636+0000 mon.smithi104 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.149:0/2886603046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:30:44.962 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:44.962 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:30:46.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:44.284780+0000 mgr.smithi104.ohwbhf (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:46.644 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.640+0000 7fe60c8e5700 1 -- 172.21.15.149:0/24418712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6080f5410 msgr2=0x7fe6080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:46.644 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.640+0000 7fe60c8e5700 1 --2- 172.21.15.149:0/24418712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6080f5410 0x7fe6080f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe5f0009fb0 tx=0x7fe5f0009b30).stop 2024-02-12T16:30:46.644 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.640+0000 7fe60c8e5700 1 -- 172.21.15.149:0/24418712 shutdown_connections 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.640+0000 7fe60c8e5700 1 --2- 172.21.15.149:0/24418712 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6080f5410 0x7fe6080f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.640+0000 7fe60c8e5700 1 -- 172.21.15.149:0/24418712 >> 172.21.15.149:0/24418712 conn(0x7fe6080f0b70 msgr2=0x7fe6080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.641+0000 7fe60c8e5700 1 -- 172.21.15.149:0/24418712 shutdown_connections 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.641+0000 7fe60c8e5700 1 -- 172.21.15.149:0/24418712 wait complete. 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.641+0000 7fe60c8e5700 1 Processor -- start 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.641+0000 7fe60c8e5700 1 -- start start 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.641+0000 7fe60c8e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6080f5410 0x7fe608109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.641+0000 7fe60c8e5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe608060220 con 0x7fe6080f5410 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.641+0000 7fe60659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6080f5410 0x7fe608109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.641+0000 7fe60659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6080f5410 0x7fe608109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:40248/0 (socket says 172.21.15.149:40248) 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.641+0000 7fe60659c700 1 -- 172.21.15.149:0/2560432948 learned_addr learned my addr 172.21.15.149:0/2560432948 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.642+0000 7fe60659c700 1 -- 172.21.15.149:0/2560432948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f0005040 con 0x7fe6080f5410 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.642+0000 7fe60659c700 1 --2- 172.21.15.149:0/2560432948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6080f5410 0x7fe608109500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fe5f0005010 tx=0x7fe5f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.642+0000 7fe5feffd700 1 -- 172.21.15.149:0/2560432948 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5f0009320 con 0x7fe6080f5410 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.642+0000 7fe60c8e5700 1 -- 172.21.15.149:0/2560432948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe608060420 con 0x7fe6080f5410 2024-02-12T16:30:46.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.642+0000 7fe5feffd700 1 -- 172.21.15.149:0/2560432948 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5f0013380 con 0x7fe6080f5410 2024-02-12T16:30:46.646 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.642+0000 7fe5feffd700 1 -- 172.21.15.149:0/2560432948 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5f0014e50 con 0x7fe6080f5410 2024-02-12T16:30:46.646 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.642+0000 7fe60c8e5700 1 -- 172.21.15.149:0/2560432948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe608109bb0 con 0x7fe6080f5410 2024-02-12T16:30:46.646 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.643+0000 7fe5feffd700 1 -- 172.21.15.149:0/2560432948 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5f0014420 con 0x7fe6080f5410 2024-02-12T16:30:46.647 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.643+0000 7fe60c8e5700 1 -- 172.21.15.149:0/2560432948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe608059100 con 0x7fe6080f5410 2024-02-12T16:30:46.647 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.643+0000 7fe5feffd700 1 --2- 172.21.15.149:0/2560432948 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe5f402ff10 0x7fe5f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:46.647 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.643+0000 7fe5feffd700 1 -- 172.21.15.149:0/2560432948 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe5f0040f40 con 0x7fe6080f5410 2024-02-12T16:30:46.647 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.643+0000 7fe605d9b700 1 --2- 172.21.15.149:0/2560432948 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe5f402ff10 0x7fe5f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:46.647 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.644+0000 7fe605d9b700 1 --2- 172.21.15.149:0/2560432948 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe5f402ff10 0x7fe5f40323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe5f80096a0 tx=0x7fe5f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:46.649 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.648+0000 7fe5feffd700 1 -- 172.21.15.149:0/2560432948 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5f00095f0 con 0x7fe6080f5410 2024-02-12T16:30:46.856 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.854+0000 7fe60c8e5700 1 -- 172.21.15.149:0/2560432948 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe608045ad0 con 0x7fe6080f5410 2024-02-12T16:30:46.856 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.854+0000 7fe5feffd700 1 -- 172.21.15.149:0/2560432948 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe5f001db70 con 0x7fe6080f5410 2024-02-12T16:30:46.856 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:46.857 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:30:46.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.856+0000 7fe60c8e5700 1 -- 172.21.15.149:0/2560432948 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe5f402ff10 msgr2=0x7fe5f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:46.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.856+0000 7fe60c8e5700 1 --2- 172.21.15.149:0/2560432948 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe5f402ff10 0x7fe5f40323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe5f80096a0 tx=0x7fe5f8006b40).stop 2024-02-12T16:30:46.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.856+0000 7fe60c8e5700 1 -- 172.21.15.149:0/2560432948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6080f5410 msgr2=0x7fe608109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:46.858 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.856+0000 7fe60c8e5700 1 --2- 172.21.15.149:0/2560432948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6080f5410 0x7fe608109500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fe5f0005010 tx=0x7fe5f000b040).stop 2024-02-12T16:30:46.859 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.856+0000 7fe60c8e5700 1 -- 172.21.15.149:0/2560432948 shutdown_connections 2024-02-12T16:30:46.859 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.856+0000 7fe60c8e5700 1 --2- 172.21.15.149:0/2560432948 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe5f402ff10 0x7fe5f40323d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:46.859 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.856+0000 7fe60c8e5700 1 --2- 172.21.15.149:0/2560432948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe6080f5410 0x7fe608109500 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:46.859 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.856+0000 7fe60c8e5700 1 -- 172.21.15.149:0/2560432948 >> 172.21.15.149:0/2560432948 conn(0x7fe6080f0b70 msgr2=0x7fe6080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:46.859 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.856+0000 7fe60c8e5700 1 -- 172.21.15.149:0/2560432948 shutdown_connections 2024-02-12T16:30:46.859 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:46.856+0000 7fe60c8e5700 1 -- 172.21.15.149:0/2560432948 wait complete. 2024-02-12T16:30:46.859 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:30:47.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:47 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:46.855497+0000 mon.smithi104 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.149:0/2560432948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:30:48.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:48.381 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:30:48.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:46.285032+0000 mgr.smithi104.ohwbhf (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:50.178 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.175+0000 7f3df07b0700 1 -- 172.21.15.149:0/809207616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3de80f6510 msgr2=0x7f3de80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:50.178 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.175+0000 7f3df07b0700 1 --2- 172.21.15.149:0/809207616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3de80f6510 0x7f3de80f6930 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f3dd8009fb0 tx=0x7f3dd8009b30).stop 2024-02-12T16:30:50.178 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.175+0000 7f3df07b0700 1 -- 172.21.15.149:0/809207616 shutdown_connections 2024-02-12T16:30:50.178 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.175+0000 7f3df07b0700 1 --2- 172.21.15.149:0/809207616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3de80f6510 0x7f3de80f6930 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:50.178 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.175+0000 7f3df07b0700 1 -- 172.21.15.149:0/809207616 >> 172.21.15.149:0/809207616 conn(0x7f3de80f1c70 msgr2=0x7f3de80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:50.178 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.175+0000 7f3df07b0700 1 -- 172.21.15.149:0/809207616 shutdown_connections 2024-02-12T16:30:50.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.175+0000 7f3df07b0700 1 -- 172.21.15.149:0/809207616 wait complete. 2024-02-12T16:30:50.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.176+0000 7f3df07b0700 1 Processor -- start 2024-02-12T16:30:50.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.176+0000 7f3df07b0700 1 -- start start 2024-02-12T16:30:50.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.176+0000 7f3df07b0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3de80f6510 0x7f3de81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:50.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.176+0000 7f3df07b0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de8106cd0 con 0x7f3de80f6510 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.176+0000 7f3dee54c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3de80f6510 0x7f3de81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.176+0000 7f3dee54c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3de80f6510 0x7f3de81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:40266/0 (socket says 172.21.15.149:40266) 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.176+0000 7f3dee54c700 1 -- 172.21.15.149:0/1309764939 learned_addr learned my addr 172.21.15.149:0/1309764939 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.177+0000 7f3dee54c700 1 -- 172.21.15.149:0/1309764939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd8005040 con 0x7f3de80f6510 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.177+0000 7f3dee54c700 1 --2- 172.21.15.149:0/1309764939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3de80f6510 0x7f3de81067e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f3dd80089f0 tx=0x7f3dd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.177+0000 7f3ddeffd700 1 -- 172.21.15.149:0/1309764939 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dd8009320 con 0x7f3de80f6510 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.177+0000 7f3df07b0700 1 -- 172.21.15.149:0/1309764939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de8106ed0 con 0x7f3de80f6510 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.177+0000 7f3ddeffd700 1 -- 172.21.15.149:0/1309764939 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3dd8013380 con 0x7f3de80f6510 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.177+0000 7f3ddeffd700 1 -- 172.21.15.149:0/1309764939 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dd801ed80 con 0x7f3de80f6510 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.177+0000 7f3df07b0700 1 -- 172.21.15.149:0/1309764939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de8107360 con 0x7f3de80f6510 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.178+0000 7f3ddeffd700 1 -- 172.21.15.149:0/1309764939 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3dd801e420 con 0x7f3de80f6510 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.178+0000 7f3df07b0700 1 -- 172.21.15.149:0/1309764939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de8045ad0 con 0x7f3de80f6510 2024-02-12T16:30:50.181 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.178+0000 7f3ddeffd700 1 --2- 172.21.15.149:0/1309764939 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3dd402ff60 0x7f3dd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:50.182 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.178+0000 7f3ddeffd700 1 -- 172.21.15.149:0/1309764939 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3dd8041030 con 0x7f3de80f6510 2024-02-12T16:30:50.182 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.178+0000 7f3dedd4b700 1 --2- 172.21.15.149:0/1309764939 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3dd402ff60 0x7f3dd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:50.182 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.179+0000 7f3dedd4b700 1 --2- 172.21.15.149:0/1309764939 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3dd402ff60 0x7f3dd4032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f3de40096a0 tx=0x7f3de4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:50.185 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.183+0000 7f3ddeffd700 1 -- 172.21.15.149:0/1309764939 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3dd801ccf0 con 0x7f3de80f6510 2024-02-12T16:30:50.394 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.392+0000 7f3df07b0700 1 -- 172.21.15.149:0/1309764939 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3de8108210 con 0x7f3de80f6510 2024-02-12T16:30:50.394 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.392+0000 7f3ddeffd700 1 -- 172.21.15.149:0/1309764939 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3dd801cb10 con 0x7f3de80f6510 2024-02-12T16:30:50.395 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:50.395 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:30:50.397 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.394+0000 7f3df07b0700 1 -- 172.21.15.149:0/1309764939 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3dd402ff60 msgr2=0x7f3dd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:50.397 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.394+0000 7f3df07b0700 1 --2- 172.21.15.149:0/1309764939 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3dd402ff60 0x7f3dd4032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f3de40096a0 tx=0x7f3de4006b40).stop 2024-02-12T16:30:50.397 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.394+0000 7f3df07b0700 1 -- 172.21.15.149:0/1309764939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3de80f6510 msgr2=0x7f3de81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:50.397 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.394+0000 7f3df07b0700 1 --2- 172.21.15.149:0/1309764939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3de80f6510 0x7f3de81067e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f3dd80089f0 tx=0x7f3dd800b040).stop 2024-02-12T16:30:50.397 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.394+0000 7f3df07b0700 1 -- 172.21.15.149:0/1309764939 shutdown_connections 2024-02-12T16:30:50.397 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.394+0000 7f3df07b0700 1 --2- 172.21.15.149:0/1309764939 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3dd402ff60 0x7f3dd4032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:50.397 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.394+0000 7f3df07b0700 1 --2- 172.21.15.149:0/1309764939 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3de80f6510 0x7f3de81067e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:50.397 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.394+0000 7f3df07b0700 1 -- 172.21.15.149:0/1309764939 >> 172.21.15.149:0/1309764939 conn(0x7f3de80f1c70 msgr2=0x7f3de80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:50.397 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.394+0000 7f3df07b0700 1 -- 172.21.15.149:0/1309764939 shutdown_connections 2024-02-12T16:30:50.397 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:50.394+0000 7f3df07b0700 1 -- 172.21.15.149:0/1309764939 wait complete. 2024-02-12T16:30:50.397 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:30:50.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:48.285327+0000 mgr.smithi104.ohwbhf (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:51.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:51 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:50.393536+0000 mon.smithi104 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.149:0/1309764939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:30:52.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:52.121 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:30:52.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:50.285688+0000 mgr.smithi104.ohwbhf (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:53.818 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.806+0000 7ff2e0734700 1 -- 172.21.15.149:0/861635073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2d80f5410 msgr2=0x7ff2d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:53.818 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.806+0000 7ff2e0734700 1 --2- 172.21.15.149:0/861635073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2d80f5410 0x7ff2d80f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff2c8009fb0 tx=0x7ff2c8009b30).stop 2024-02-12T16:30:53.818 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.806+0000 7ff2e0734700 1 -- 172.21.15.149:0/861635073 shutdown_connections 2024-02-12T16:30:53.818 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.806+0000 7ff2e0734700 1 --2- 172.21.15.149:0/861635073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2d80f5410 0x7ff2d80f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:53.818 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.806+0000 7ff2e0734700 1 -- 172.21.15.149:0/861635073 >> 172.21.15.149:0/861635073 conn(0x7ff2d80f0b70 msgr2=0x7ff2d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.806+0000 7ff2e0734700 1 -- 172.21.15.149:0/861635073 shutdown_connections 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.806+0000 7ff2e0734700 1 -- 172.21.15.149:0/861635073 wait complete. 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.807+0000 7ff2e0734700 1 Processor -- start 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.807+0000 7ff2e0734700 1 -- start start 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.807+0000 7ff2e0734700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2d80f5410 0x7ff2d81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.807+0000 7ff2e0734700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d8105bd0 con 0x7ff2d80f5410 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.807+0000 7ff2de4d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2d80f5410 0x7ff2d81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.807+0000 7ff2de4d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2d80f5410 0x7ff2d81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:40288/0 (socket says 172.21.15.149:40288) 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.807+0000 7ff2de4d0700 1 -- 172.21.15.149:0/1372678314 learned_addr learned my addr 172.21.15.149:0/1372678314 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.808+0000 7ff2de4d0700 1 -- 172.21.15.149:0/1372678314 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c8005040 con 0x7ff2d80f5410 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.808+0000 7ff2de4d0700 1 --2- 172.21.15.149:0/1372678314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2d80f5410 0x7ff2d81056e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff2c8005010 tx=0x7ff2c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.808+0000 7ff2ceffd700 1 -- 172.21.15.149:0/1372678314 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2c8009320 con 0x7ff2d80f5410 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.808+0000 7ff2e0734700 1 -- 172.21.15.149:0/1372678314 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d8105dd0 con 0x7ff2d80f5410 2024-02-12T16:30:53.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.808+0000 7ff2ceffd700 1 -- 172.21.15.149:0/1372678314 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2c8013380 con 0x7ff2d80f5410 2024-02-12T16:30:53.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.808+0000 7ff2ceffd700 1 -- 172.21.15.149:0/1372678314 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2c8014e50 con 0x7ff2d80f5410 2024-02-12T16:30:53.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.808+0000 7ff2e0734700 1 -- 172.21.15.149:0/1372678314 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d8106260 con 0x7ff2d80f5410 2024-02-12T16:30:53.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.809+0000 7ff2ceffd700 1 -- 172.21.15.149:0/1372678314 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2c80095f0 con 0x7ff2d80f5410 2024-02-12T16:30:53.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.809+0000 7ff2e0734700 1 -- 172.21.15.149:0/1372678314 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d8045ad0 con 0x7ff2d80f5410 2024-02-12T16:30:53.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.809+0000 7ff2ceffd700 1 --2- 172.21.15.149:0/1372678314 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff2c402ff10 0x7ff2c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:53.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.809+0000 7ff2ceffd700 1 -- 172.21.15.149:0/1372678314 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2c8040f40 con 0x7ff2d80f5410 2024-02-12T16:30:53.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.809+0000 7ff2ddccf700 1 --2- 172.21.15.149:0/1372678314 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff2c402ff10 0x7ff2c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:53.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.810+0000 7ff2ddccf700 1 --2- 172.21.15.149:0/1372678314 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff2c402ff10 0x7ff2c40323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff2d40096a0 tx=0x7ff2d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:53.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:53.814+0000 7ff2ceffd700 1 -- 172.21.15.149:0/1372678314 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2c8014420 con 0x7ff2d80f5410 2024-02-12T16:30:54.023 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.021+0000 7ff2e0734700 1 -- 172.21.15.149:0/1372678314 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2d81070b0 con 0x7ff2d80f5410 2024-02-12T16:30:54.023 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.021+0000 7ff2ceffd700 1 -- 172.21.15.149:0/1372678314 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff2c8010230 con 0x7ff2d80f5410 2024-02-12T16:30:54.023 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:54.024 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:30:54.025 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.023+0000 7ff2e0734700 1 -- 172.21.15.149:0/1372678314 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff2c402ff10 msgr2=0x7ff2c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:54.025 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.023+0000 7ff2e0734700 1 --2- 172.21.15.149:0/1372678314 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff2c402ff10 0x7ff2c40323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff2d40096a0 tx=0x7ff2d4006b40).stop 2024-02-12T16:30:54.025 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.023+0000 7ff2e0734700 1 -- 172.21.15.149:0/1372678314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2d80f5410 msgr2=0x7ff2d81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:54.025 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.023+0000 7ff2e0734700 1 --2- 172.21.15.149:0/1372678314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2d80f5410 0x7ff2d81056e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff2c8005010 tx=0x7ff2c800b040).stop 2024-02-12T16:30:54.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.023+0000 7ff2e0734700 1 -- 172.21.15.149:0/1372678314 shutdown_connections 2024-02-12T16:30:54.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.023+0000 7ff2e0734700 1 --2- 172.21.15.149:0/1372678314 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff2c402ff10 0x7ff2c40323d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:54.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.023+0000 7ff2e0734700 1 --2- 172.21.15.149:0/1372678314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2d80f5410 0x7ff2d81056e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:54.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.023+0000 7ff2e0734700 1 -- 172.21.15.149:0/1372678314 >> 172.21.15.149:0/1372678314 conn(0x7ff2d80f0b70 msgr2=0x7ff2d80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:54.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.023+0000 7ff2e0734700 1 -- 172.21.15.149:0/1372678314 shutdown_connections 2024-02-12T16:30:54.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:54.023+0000 7ff2e0734700 1 -- 172.21.15.149:0/1372678314 wait complete. 2024-02-12T16:30:54.026 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:30:54.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:52.285988+0000 mgr.smithi104.ohwbhf (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:54.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:54.022588+0000 mon.smithi104 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.149:0/1372678314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:30:55.666 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:55.666 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:30:56.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:54.286321+0000 mgr.smithi104.ohwbhf (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:57.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.410+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/1692002691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a780601e0 msgr2=0x7f5a780f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:57.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.410+0000 7f5a7c9a9700 1 --2- 172.21.15.149:0/1692002691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a780601e0 0x7f5a780f8f20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5a68009fb0 tx=0x7f5a68009b30).stop 2024-02-12T16:30:57.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.410+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/1692002691 shutdown_connections 2024-02-12T16:30:57.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.410+0000 7f5a7c9a9700 1 --2- 172.21.15.149:0/1692002691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a780601e0 0x7f5a780f8f20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:57.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.410+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/1692002691 >> 172.21.15.149:0/1692002691 conn(0x7f5a780f09a0 msgr2=0x7f5a780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:57.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.410+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/1692002691 shutdown_connections 2024-02-12T16:30:57.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.410+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/1692002691 wait complete. 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.410+0000 7f5a7c9a9700 1 Processor -- start 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.411+0000 7f5a7c9a9700 1 -- start start 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.411+0000 7f5a7c9a9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a780601e0 0x7f5a78106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.411+0000 7f5a7c9a9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a78106af0 con 0x7f5a780601e0 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.411+0000 7f5a777fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a780601e0 0x7f5a78106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.411+0000 7f5a777fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a780601e0 0x7f5a78106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55294/0 (socket says 172.21.15.149:55294) 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.411+0000 7f5a777fe700 1 -- 172.21.15.149:0/4244947263 learned_addr learned my addr 172.21.15.149:0/4244947263 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.411+0000 7f5a777fe700 1 -- 172.21.15.149:0/4244947263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a68005040 con 0x7f5a780601e0 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.411+0000 7f5a777fe700 1 --2- 172.21.15.149:0/4244947263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a780601e0 0x7f5a78106600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f5a68005010 tx=0x7f5a6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.412+0000 7f5a5ffff700 1 -- 172.21.15.149:0/4244947263 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a68009320 con 0x7f5a780601e0 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.412+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/4244947263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a78106cf0 con 0x7f5a780601e0 2024-02-12T16:30:57.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.412+0000 7f5a5ffff700 1 -- 172.21.15.149:0/4244947263 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a68013380 con 0x7f5a780601e0 2024-02-12T16:30:57.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.412+0000 7f5a5ffff700 1 -- 172.21.15.149:0/4244947263 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a68014e50 con 0x7f5a780601e0 2024-02-12T16:30:57.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.412+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/4244947263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a78107180 con 0x7f5a780601e0 2024-02-12T16:30:57.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.412+0000 7f5a5ffff700 1 -- 172.21.15.149:0/4244947263 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a680263a0 con 0x7f5a780601e0 2024-02-12T16:30:57.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.413+0000 7f5a5ffff700 1 --2- 172.21.15.149:0/4244947263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a6002ff10 0x7f5a600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:30:57.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.413+0000 7f5a5ffff700 1 -- 172.21.15.149:0/4244947263 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a68040f90 con 0x7f5a780601e0 2024-02-12T16:30:57.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.413+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/4244947263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a78045ad0 con 0x7f5a780601e0 2024-02-12T16:30:57.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.413+0000 7f5a76ffd700 1 --2- 172.21.15.149:0/4244947263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a6002ff10 0x7f5a600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:30:57.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.413+0000 7f5a76ffd700 1 --2- 172.21.15.149:0/4244947263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a6002ff10 0x7f5a600323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f5a6c0096a0 tx=0x7f5a6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:30:57.418 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.417+0000 7f5a5ffff700 1 -- 172.21.15.149:0/4244947263 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a68012920 con 0x7f5a780601e0 2024-02-12T16:30:57.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.622+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/4244947263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a78107ed0 con 0x7f5a780601e0 2024-02-12T16:30:57.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.622+0000 7f5a5ffff700 1 -- 172.21.15.149:0/4244947263 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5a6802b020 con 0x7f5a780601e0 2024-02-12T16:30:57.624 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:30:57.624 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:30:57.625 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.624+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/4244947263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a6002ff10 msgr2=0x7f5a600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:57.625 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.624+0000 7f5a7c9a9700 1 --2- 172.21.15.149:0/4244947263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a6002ff10 0x7f5a600323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f5a6c0096a0 tx=0x7f5a6c006b40).stop 2024-02-12T16:30:57.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.624+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/4244947263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a780601e0 msgr2=0x7f5a78106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:30:57.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.624+0000 7f5a7c9a9700 1 --2- 172.21.15.149:0/4244947263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a780601e0 0x7f5a78106600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f5a68005010 tx=0x7f5a6800b040).stop 2024-02-12T16:30:57.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.624+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/4244947263 shutdown_connections 2024-02-12T16:30:57.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.624+0000 7f5a7c9a9700 1 --2- 172.21.15.149:0/4244947263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5a6002ff10 0x7f5a600323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:57.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.624+0000 7f5a7c9a9700 1 --2- 172.21.15.149:0/4244947263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a780601e0 0x7f5a78106600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:30:57.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.624+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/4244947263 >> 172.21.15.149:0/4244947263 conn(0x7f5a780f09a0 msgr2=0x7f5a780f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:30:57.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.624+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/4244947263 shutdown_connections 2024-02-12T16:30:57.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:30:57.624+0000 7f5a7c9a9700 1 -- 172.21.15.149:0/4244947263 wait complete. 2024-02-12T16:30:57.626 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:30:58.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:56.286635+0000 mgr.smithi104.ohwbhf (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:30:58.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:30:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:30:57.622618+0000 mon.smithi104 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.149:0/4244947263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:30:59.204 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:30:59.204 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:00.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:30:58.286869+0000 mgr.smithi104.ohwbhf (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:00.958 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.957+0000 7f695966b700 1 -- 172.21.15.149:0/3378671090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69540f51a0 msgr2=0x7f69540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.957+0000 7f695966b700 1 --2- 172.21.15.149:0/3378671090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69540f51a0 0x7f69540f55c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f6944009fb0 tx=0x7f6944009b30).stop 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.957+0000 7f695966b700 1 -- 172.21.15.149:0/3378671090 shutdown_connections 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.957+0000 7f695966b700 1 --2- 172.21.15.149:0/3378671090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69540f51a0 0x7f69540f55c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.957+0000 7f695966b700 1 -- 172.21.15.149:0/3378671090 >> 172.21.15.149:0/3378671090 conn(0x7f69540f09a0 msgr2=0x7f69540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.957+0000 7f695966b700 1 -- 172.21.15.149:0/3378671090 shutdown_connections 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.957+0000 7f695966b700 1 -- 172.21.15.149:0/3378671090 wait complete. 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.957+0000 7f695966b700 1 Processor -- start 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.957+0000 7f695966b700 1 -- start start 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f695966b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69540f51a0 0x7f69541054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f695966b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69541059c0 con 0x7f69540f51a0 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f6953fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69540f51a0 0x7f69541054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f6953fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69540f51a0 0x7f69541054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55298/0 (socket says 172.21.15.149:55298) 2024-02-12T16:31:00.959 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f6953fff700 1 -- 172.21.15.149:0/2708528474 learned_addr learned my addr 172.21.15.149:0/2708528474 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:00.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f6953fff700 1 -- 172.21.15.149:0/2708528474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944005040 con 0x7f69540f51a0 2024-02-12T16:31:00.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f6953fff700 1 --2- 172.21.15.149:0/2708528474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69540f51a0 0x7f69541054d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f69440089f0 tx=0x7f694400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:00.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f6950ff9700 1 -- 172.21.15.149:0/2708528474 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6944009320 con 0x7f69540f51a0 2024-02-12T16:31:00.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f695966b700 1 -- 172.21.15.149:0/2708528474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6954105bc0 con 0x7f69540f51a0 2024-02-12T16:31:00.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f6950ff9700 1 -- 172.21.15.149:0/2708528474 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6944013380 con 0x7f69540f51a0 2024-02-12T16:31:00.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f6950ff9700 1 -- 172.21.15.149:0/2708528474 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6944014e20 con 0x7f69540f51a0 2024-02-12T16:31:00.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.958+0000 7f695966b700 1 -- 172.21.15.149:0/2708528474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6954106050 con 0x7f69540f51a0 2024-02-12T16:31:00.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.959+0000 7f6950ff9700 1 -- 172.21.15.149:0/2708528474 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6944014420 con 0x7f69540f51a0 2024-02-12T16:31:00.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.959+0000 7f695966b700 1 -- 172.21.15.149:0/2708528474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6954045ad0 con 0x7f69540f51a0 2024-02-12T16:31:00.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.959+0000 7f6950ff9700 1 --2- 172.21.15.149:0/2708528474 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f693c032120 0x7f693c0345e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:00.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.959+0000 7f6950ff9700 1 -- 172.21.15.149:0/2708528474 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6944040f00 con 0x7f69540f51a0 2024-02-12T16:31:00.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.960+0000 7f69537fe700 1 --2- 172.21.15.149:0/2708528474 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f693c032120 0x7f693c0345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:00.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.960+0000 7f69537fe700 1 --2- 172.21.15.149:0/2708528474 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f693c032120 0x7f693c0345e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f69480096a0 tx=0x7f6948006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:00.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:00.964+0000 7f6950ff9700 1 -- 172.21.15.149:0/2708528474 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69440095f0 con 0x7f69540f51a0 2024-02-12T16:31:01.170 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.168+0000 7f695966b700 1 -- 172.21.15.149:0/2708528474 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6954106f00 con 0x7f69540f51a0 2024-02-12T16:31:01.170 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.169+0000 7f6950ff9700 1 -- 172.21.15.149:0/2708528474 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f694401db70 con 0x7f69540f51a0 2024-02-12T16:31:01.170 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:01.171 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:01.172 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.171+0000 7f695966b700 1 -- 172.21.15.149:0/2708528474 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f693c032120 msgr2=0x7f693c0345e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:01.172 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.171+0000 7f695966b700 1 --2- 172.21.15.149:0/2708528474 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f693c032120 0x7f693c0345e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f69480096a0 tx=0x7f6948006b40).stop 2024-02-12T16:31:01.172 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.171+0000 7f695966b700 1 -- 172.21.15.149:0/2708528474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69540f51a0 msgr2=0x7f69541054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:01.172 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.171+0000 7f695966b700 1 --2- 172.21.15.149:0/2708528474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69540f51a0 0x7f69541054d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f69440089f0 tx=0x7f694400b040).stop 2024-02-12T16:31:01.172 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.171+0000 7f695966b700 1 -- 172.21.15.149:0/2708528474 shutdown_connections 2024-02-12T16:31:01.172 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.171+0000 7f695966b700 1 --2- 172.21.15.149:0/2708528474 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f693c032120 0x7f693c0345e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:01.172 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.171+0000 7f695966b700 1 --2- 172.21.15.149:0/2708528474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69540f51a0 0x7f69541054d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.171+0000 7f695966b700 1 -- 172.21.15.149:0/2708528474 >> 172.21.15.149:0/2708528474 conn(0x7f69540f09a0 msgr2=0x7f69540f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.171+0000 7f695966b700 1 -- 172.21.15.149:0/2708528474 shutdown_connections 2024-02-12T16:31:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:01.171+0000 7f695966b700 1 -- 172.21.15.149:0/2708528474 wait complete. 2024-02-12T16:31:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:02.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:00.287191+0000 mgr.smithi104.ohwbhf (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:02.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:01.169392+0000 mon.smithi104 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.149:0/2708528474' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:02.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:02.855 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:04.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:02.287457+0000 mgr.smithi104.ohwbhf (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:04.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.689+0000 7f709f6fa700 1 -- 172.21.15.149:0/320019110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70980f6510 msgr2=0x7f70980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:04.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.689+0000 7f709f6fa700 1 --2- 172.21.15.149:0/320019110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70980f6510 0x7f70980f6930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f7088009fb0 tx=0x7f7088009b30).stop 2024-02-12T16:31:04.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.689+0000 7f709f6fa700 1 -- 172.21.15.149:0/320019110 shutdown_connections 2024-02-12T16:31:04.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.689+0000 7f709f6fa700 1 --2- 172.21.15.149:0/320019110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70980f6510 0x7f70980f6930 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:04.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.689+0000 7f709f6fa700 1 -- 172.21.15.149:0/320019110 >> 172.21.15.149:0/320019110 conn(0x7f70980f1c70 msgr2=0x7f70980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:04.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.689+0000 7f709f6fa700 1 -- 172.21.15.149:0/320019110 shutdown_connections 2024-02-12T16:31:04.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.689+0000 7f709f6fa700 1 -- 172.21.15.149:0/320019110 wait complete. 2024-02-12T16:31:04.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.690+0000 7f709f6fa700 1 Processor -- start 2024-02-12T16:31:04.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.690+0000 7f709f6fa700 1 -- start start 2024-02-12T16:31:04.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.690+0000 7f709f6fa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70980f6510 0x7f70981067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:04.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.690+0000 7f709f6fa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7098106cd0 con 0x7f70980f6510 2024-02-12T16:31:04.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.691+0000 7f709d496700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70980f6510 0x7f70981067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:04.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.691+0000 7f709d496700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70980f6510 0x7f70981067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:42584/0 (socket says 172.21.15.149:42584) 2024-02-12T16:31:04.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.691+0000 7f709d496700 1 -- 172.21.15.149:0/973458246 learned_addr learned my addr 172.21.15.149:0/973458246 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:04.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.691+0000 7f709d496700 1 -- 172.21.15.149:0/973458246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7088005040 con 0x7f70980f6510 2024-02-12T16:31:04.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.691+0000 7f709d496700 1 --2- 172.21.15.149:0/973458246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70980f6510 0x7f70981067e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f7088005010 tx=0x7f708800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:04.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.691+0000 7f708dffb700 1 -- 172.21.15.149:0/973458246 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7088009320 con 0x7f70980f6510 2024-02-12T16:31:04.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.691+0000 7f709f6fa700 1 -- 172.21.15.149:0/973458246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7098106ed0 con 0x7f70980f6510 2024-02-12T16:31:04.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.691+0000 7f708dffb700 1 -- 172.21.15.149:0/973458246 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7088013380 con 0x7f70980f6510 2024-02-12T16:31:04.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.691+0000 7f708dffb700 1 -- 172.21.15.149:0/973458246 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7088014e50 con 0x7f70980f6510 2024-02-12T16:31:04.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.691+0000 7f709f6fa700 1 -- 172.21.15.149:0/973458246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7098107360 con 0x7f70980f6510 2024-02-12T16:31:04.696 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.692+0000 7f708dffb700 1 -- 172.21.15.149:0/973458246 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70880263a0 con 0x7f70980f6510 2024-02-12T16:31:04.696 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.692+0000 7f708dffb700 1 --2- 172.21.15.149:0/973458246 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f708402ff10 0x7f70840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:04.696 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.692+0000 7f709f6fa700 1 -- 172.21.15.149:0/973458246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7098045ad0 con 0x7f70980f6510 2024-02-12T16:31:04.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.692+0000 7f708dffb700 1 -- 172.21.15.149:0/973458246 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7088041160 con 0x7f70980f6510 2024-02-12T16:31:04.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.693+0000 7f709cc95700 1 --2- 172.21.15.149:0/973458246 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f708402ff10 0x7f70840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:04.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.693+0000 7f709cc95700 1 --2- 172.21.15.149:0/973458246 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f708402ff10 0x7f70840323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f70940096a0 tx=0x7f7094006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:04.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.697+0000 7f708dffb700 1 -- 172.21.15.149:0/973458246 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7088012920 con 0x7f70980f6510 2024-02-12T16:31:04.904 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.902+0000 7f709f6fa700 1 -- 172.21.15.149:0/973458246 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7098108210 con 0x7f70980f6510 2024-02-12T16:31:04.905 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.903+0000 7f708dffb700 1 -- 172.21.15.149:0/973458246 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f708800f340 con 0x7f70980f6510 2024-02-12T16:31:04.905 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:04.905 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:04.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.905+0000 7f709f6fa700 1 -- 172.21.15.149:0/973458246 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f708402ff10 msgr2=0x7f70840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:04.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.905+0000 7f709f6fa700 1 --2- 172.21.15.149:0/973458246 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f708402ff10 0x7f70840323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f70940096a0 tx=0x7f7094006b40).stop 2024-02-12T16:31:04.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.905+0000 7f709f6fa700 1 -- 172.21.15.149:0/973458246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70980f6510 msgr2=0x7f70981067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:04.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.905+0000 7f709f6fa700 1 --2- 172.21.15.149:0/973458246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70980f6510 0x7f70981067e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f7088005010 tx=0x7f708800b040).stop 2024-02-12T16:31:04.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.905+0000 7f709f6fa700 1 -- 172.21.15.149:0/973458246 shutdown_connections 2024-02-12T16:31:04.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.905+0000 7f709f6fa700 1 --2- 172.21.15.149:0/973458246 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f708402ff10 0x7f70840323d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:04.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.905+0000 7f709f6fa700 1 --2- 172.21.15.149:0/973458246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70980f6510 0x7f70981067e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:04.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.905+0000 7f709f6fa700 1 -- 172.21.15.149:0/973458246 >> 172.21.15.149:0/973458246 conn(0x7f70980f1c70 msgr2=0x7f70980f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:04.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.905+0000 7f709f6fa700 1 -- 172.21.15.149:0/973458246 shutdown_connections 2024-02-12T16:31:04.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:04.905+0000 7f709f6fa700 1 -- 172.21.15.149:0/973458246 wait complete. 2024-02-12T16:31:04.908 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:05.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:04.904364+0000 mon.smithi104 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.149:0/973458246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:06.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:06.425 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:06.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:04.287740+0000 mgr.smithi104.ohwbhf (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:08.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.240+0000 7f21d7d19700 1 -- 172.21.15.149:0/3887647492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21d00f5410 msgr2=0x7f21d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.240+0000 7f21d7d19700 1 --2- 172.21.15.149:0/3887647492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21d00f5410 0x7f21d00f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f21c0009fb0 tx=0x7f21c0009b30).stop 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.240+0000 7f21d7d19700 1 -- 172.21.15.149:0/3887647492 shutdown_connections 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.240+0000 7f21d7d19700 1 --2- 172.21.15.149:0/3887647492 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21d00f5410 0x7f21d00f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.240+0000 7f21d7d19700 1 -- 172.21.15.149:0/3887647492 >> 172.21.15.149:0/3887647492 conn(0x7f21d00f0b70 msgr2=0x7f21d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.240+0000 7f21d7d19700 1 -- 172.21.15.149:0/3887647492 shutdown_connections 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.240+0000 7f21d7d19700 1 -- 172.21.15.149:0/3887647492 wait complete. 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.240+0000 7f21d7d19700 1 Processor -- start 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.241+0000 7f21d7d19700 1 -- start start 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.241+0000 7f21d7d19700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21d00f5410 0x7f21d01067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.241+0000 7f21d7d19700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d0106ca0 con 0x7f21d00f5410 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.241+0000 7f21d5ab5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21d00f5410 0x7f21d01067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:08.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.241+0000 7f21d5ab5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21d00f5410 0x7f21d01067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:42612/0 (socket says 172.21.15.149:42612) 2024-02-12T16:31:08.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.241+0000 7f21d5ab5700 1 -- 172.21.15.149:0/1039673300 learned_addr learned my addr 172.21.15.149:0/1039673300 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:08.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.241+0000 7f21d5ab5700 1 -- 172.21.15.149:0/1039673300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c0005040 con 0x7f21d00f5410 2024-02-12T16:31:08.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.241+0000 7f21d5ab5700 1 --2- 172.21.15.149:0/1039673300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21d00f5410 0x7f21d01067b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f21c0005010 tx=0x7f21c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:08.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.241+0000 7f21c67fc700 1 -- 172.21.15.149:0/1039673300 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21c0009320 con 0x7f21d00f5410 2024-02-12T16:31:08.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.242+0000 7f21d7d19700 1 -- 172.21.15.149:0/1039673300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d0106ea0 con 0x7f21d00f5410 2024-02-12T16:31:08.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.242+0000 7f21c67fc700 1 -- 172.21.15.149:0/1039673300 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21c0013380 con 0x7f21d00f5410 2024-02-12T16:31:08.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.242+0000 7f21c67fc700 1 -- 172.21.15.149:0/1039673300 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21c0014e50 con 0x7f21d00f5410 2024-02-12T16:31:08.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.242+0000 7f21d7d19700 1 -- 172.21.15.149:0/1039673300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d0107330 con 0x7f21d00f5410 2024-02-12T16:31:08.246 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.242+0000 7f21d7d19700 1 -- 172.21.15.149:0/1039673300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d0045ad0 con 0x7f21d00f5410 2024-02-12T16:31:08.246 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.243+0000 7f21c67fc700 1 -- 172.21.15.149:0/1039673300 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21c00095f0 con 0x7f21d00f5410 2024-02-12T16:31:08.246 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.243+0000 7f21c67fc700 1 --2- 172.21.15.149:0/1039673300 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21bc02ff10 0x7f21bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:08.246 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.243+0000 7f21c67fc700 1 -- 172.21.15.149:0/1039673300 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21c0040fa0 con 0x7f21d00f5410 2024-02-12T16:31:08.246 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.243+0000 7f21d52b4700 1 --2- 172.21.15.149:0/1039673300 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21bc02ff10 0x7f21bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:08.246 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.243+0000 7f21d52b4700 1 --2- 172.21.15.149:0/1039673300 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21bc02ff10 0x7f21bc0323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f21cc0096a0 tx=0x7f21cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:08.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.247+0000 7f21c67fc700 1 -- 172.21.15.149:0/1039673300 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21c0014420 con 0x7f21d00f5410 2024-02-12T16:31:08.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:06.288093+0000 mgr.smithi104.ohwbhf (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:08.454 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.452+0000 7f21d7d19700 1 -- 172.21.15.149:0/1039673300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21d00fafa0 con 0x7f21d00f5410 2024-02-12T16:31:08.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.453+0000 7f21c67fc700 1 -- 172.21.15.149:0/1039673300 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f21c0010230 con 0x7f21d00f5410 2024-02-12T16:31:08.455 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:08.455 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:08.457 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.454+0000 7f21d7d19700 1 -- 172.21.15.149:0/1039673300 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21bc02ff10 msgr2=0x7f21bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:08.457 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.454+0000 7f21d7d19700 1 --2- 172.21.15.149:0/1039673300 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21bc02ff10 0x7f21bc0323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f21cc0096a0 tx=0x7f21cc006b40).stop 2024-02-12T16:31:08.457 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.454+0000 7f21d7d19700 1 -- 172.21.15.149:0/1039673300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21d00f5410 msgr2=0x7f21d01067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:08.457 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.454+0000 7f21d7d19700 1 --2- 172.21.15.149:0/1039673300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21d00f5410 0x7f21d01067b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f21c0005010 tx=0x7f21c000b040).stop 2024-02-12T16:31:08.457 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.454+0000 7f21d7d19700 1 -- 172.21.15.149:0/1039673300 shutdown_connections 2024-02-12T16:31:08.457 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.454+0000 7f21d7d19700 1 --2- 172.21.15.149:0/1039673300 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21bc02ff10 0x7f21bc0323d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:08.457 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.454+0000 7f21d7d19700 1 --2- 172.21.15.149:0/1039673300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21d00f5410 0x7f21d01067b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:08.457 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.454+0000 7f21d7d19700 1 -- 172.21.15.149:0/1039673300 >> 172.21.15.149:0/1039673300 conn(0x7f21d00f0b70 msgr2=0x7f21d005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:08.457 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.455+0000 7f21d7d19700 1 -- 172.21.15.149:0/1039673300 shutdown_connections 2024-02-12T16:31:08.458 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:08.455+0000 7f21d7d19700 1 -- 172.21.15.149:0/1039673300 wait complete. 2024-02-12T16:31:08.458 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:09.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:08.454148+0000 mon.smithi104 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.149:0/1039673300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:10.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:10.077 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:10.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:08.288408+0000 mgr.smithi104.ohwbhf (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:11.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.803+0000 7f3371571700 1 -- 172.21.15.149:0/2139210920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336c0f5410 msgr2=0x7f336c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:11.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.803+0000 7f3371571700 1 --2- 172.21.15.149:0/2139210920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336c0f5410 0x7f336c0f5830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f3360009fb0 tx=0x7f3360009b30).stop 2024-02-12T16:31:11.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.803+0000 7f3371571700 1 -- 172.21.15.149:0/2139210920 shutdown_connections 2024-02-12T16:31:11.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.803+0000 7f3371571700 1 --2- 172.21.15.149:0/2139210920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336c0f5410 0x7f336c0f5830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:11.806 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.803+0000 7f3371571700 1 -- 172.21.15.149:0/2139210920 >> 172.21.15.149:0/2139210920 conn(0x7f336c0f0b70 msgr2=0x7f336c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:11.806 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.803+0000 7f3371571700 1 -- 172.21.15.149:0/2139210920 shutdown_connections 2024-02-12T16:31:11.806 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.803+0000 7f3371571700 1 -- 172.21.15.149:0/2139210920 wait complete. 2024-02-12T16:31:11.808 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.804+0000 7f3371571700 1 Processor -- start 2024-02-12T16:31:11.808 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.804+0000 7f3371571700 1 -- start start 2024-02-12T16:31:11.808 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.804+0000 7f3371571700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336c0f5410 0x7f336c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:11.808 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.804+0000 7f3371571700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c106cc0 con 0x7f336c0f5410 2024-02-12T16:31:11.808 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.804+0000 7f336affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336c0f5410 0x7f336c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:11.808 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.804+0000 7f336affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336c0f5410 0x7f336c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:42638/0 (socket says 172.21.15.149:42638) 2024-02-12T16:31:11.808 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.804+0000 7f336affd700 1 -- 172.21.15.149:0/4066647337 learned_addr learned my addr 172.21.15.149:0/4066647337 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:11.808 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.804+0000 7f336affd700 1 -- 172.21.15.149:0/4066647337 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360005040 con 0x7f336c0f5410 2024-02-12T16:31:11.808 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.805+0000 7f336affd700 1 --2- 172.21.15.149:0/4066647337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336c0f5410 0x7f336c1067d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f3360005010 tx=0x7f336000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:11.808 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.805+0000 7f33537fe700 1 -- 172.21.15.149:0/4066647337 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3360009320 con 0x7f336c0f5410 2024-02-12T16:31:11.809 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.805+0000 7f3371571700 1 -- 172.21.15.149:0/4066647337 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c106ec0 con 0x7f336c0f5410 2024-02-12T16:31:11.809 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.805+0000 7f33537fe700 1 -- 172.21.15.149:0/4066647337 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3360013380 con 0x7f336c0f5410 2024-02-12T16:31:11.809 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.805+0000 7f33537fe700 1 -- 172.21.15.149:0/4066647337 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3360014e50 con 0x7f336c0f5410 2024-02-12T16:31:11.809 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.805+0000 7f3371571700 1 -- 172.21.15.149:0/4066647337 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c107350 con 0x7f336c0f5410 2024-02-12T16:31:11.809 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.805+0000 7f33537fe700 1 -- 172.21.15.149:0/4066647337 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3360014420 con 0x7f336c0f5410 2024-02-12T16:31:11.809 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.806+0000 7f3371571700 1 -- 172.21.15.149:0/4066647337 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336c045ad0 con 0x7f336c0f5410 2024-02-12T16:31:11.809 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.806+0000 7f33537fe700 1 --2- 172.21.15.149:0/4066647337 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f335402ff10 0x7f33540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:11.809 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.806+0000 7f33537fe700 1 -- 172.21.15.149:0/4066647337 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3360040e30 con 0x7f336c0f5410 2024-02-12T16:31:11.809 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.806+0000 7f336a7fc700 1 --2- 172.21.15.149:0/4066647337 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f335402ff10 0x7f33540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:11.809 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.806+0000 7f336a7fc700 1 --2- 172.21.15.149:0/4066647337 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f335402ff10 0x7f33540323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f335c0096a0 tx=0x7f335c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:11.812 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:11.810+0000 7f33537fe700 1 -- 172.21.15.149:0/4066647337 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33600095f0 con 0x7f336c0f5410 2024-02-12T16:31:12.018 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.016+0000 7f3371571700 1 -- 172.21.15.149:0/4066647337 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f336c108200 con 0x7f336c0f5410 2024-02-12T16:31:12.019 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.016+0000 7f33537fe700 1 -- 172.21.15.149:0/4066647337 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f336001db70 con 0x7f336c0f5410 2024-02-12T16:31:12.019 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:12.019 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:12.020 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.018+0000 7f3371571700 1 -- 172.21.15.149:0/4066647337 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f335402ff10 msgr2=0x7f33540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:12.020 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.018+0000 7f3371571700 1 --2- 172.21.15.149:0/4066647337 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f335402ff10 0x7f33540323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f335c0096a0 tx=0x7f335c006b40).stop 2024-02-12T16:31:12.020 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.018+0000 7f3371571700 1 -- 172.21.15.149:0/4066647337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336c0f5410 msgr2=0x7f336c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:12.020 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.018+0000 7f3371571700 1 --2- 172.21.15.149:0/4066647337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336c0f5410 0x7f336c1067d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f3360005010 tx=0x7f336000b040).stop 2024-02-12T16:31:12.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.018+0000 7f3371571700 1 -- 172.21.15.149:0/4066647337 shutdown_connections 2024-02-12T16:31:12.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.018+0000 7f3371571700 1 --2- 172.21.15.149:0/4066647337 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f335402ff10 0x7f33540323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:12.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.018+0000 7f3371571700 1 --2- 172.21.15.149:0/4066647337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f336c0f5410 0x7f336c1067d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:12.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.018+0000 7f3371571700 1 -- 172.21.15.149:0/4066647337 >> 172.21.15.149:0/4066647337 conn(0x7f336c0f0b70 msgr2=0x7f336c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:12.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.018+0000 7f3371571700 1 -- 172.21.15.149:0/4066647337 shutdown_connections 2024-02-12T16:31:12.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:12.018+0000 7f3371571700 1 -- 172.21.15.149:0/4066647337 wait complete. 2024-02-12T16:31:12.021 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:12.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:10.288778+0000 mgr.smithi104.ohwbhf (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:12.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:12.017632+0000 mon.smithi104 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.149:0/4066647337' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:13.692 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:13.693 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:14.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:12.289152+0000 mgr.smithi104.ohwbhf (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:15.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.512+0000 7fafe5f76700 1 -- 172.21.15.149:0/861179251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe00f5410 msgr2=0x7fafe00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:15.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.512+0000 7fafe5f76700 1 --2- 172.21.15.149:0/861179251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe00f5410 0x7fafe00f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fafd0009fb0 tx=0x7fafd0009b30).stop 2024-02-12T16:31:15.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.513+0000 7fafe5f76700 1 -- 172.21.15.149:0/861179251 shutdown_connections 2024-02-12T16:31:15.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.513+0000 7fafe5f76700 1 --2- 172.21.15.149:0/861179251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe00f5410 0x7fafe00f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:15.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.513+0000 7fafe5f76700 1 -- 172.21.15.149:0/861179251 >> 172.21.15.149:0/861179251 conn(0x7fafe00f0b70 msgr2=0x7fafe00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:15.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.513+0000 7fafe5f76700 1 -- 172.21.15.149:0/861179251 shutdown_connections 2024-02-12T16:31:15.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.513+0000 7fafe5f76700 1 -- 172.21.15.149:0/861179251 wait complete. 2024-02-12T16:31:15.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.513+0000 7fafe5f76700 1 Processor -- start 2024-02-12T16:31:15.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.513+0000 7fafe5f76700 1 -- start start 2024-02-12T16:31:15.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.513+0000 7fafe5f76700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe00f5410 0x7fafe0104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:15.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.513+0000 7fafe5f76700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe0104b00 con 0x7fafe00f5410 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.514+0000 7fafdf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe00f5410 0x7fafe0104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.514+0000 7fafdf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe00f5410 0x7fafe0104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:45276/0 (socket says 172.21.15.149:45276) 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.514+0000 7fafdf7fe700 1 -- 172.21.15.149:0/4001791057 learned_addr learned my addr 172.21.15.149:0/4001791057 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.514+0000 7fafdf7fe700 1 -- 172.21.15.149:0/4001791057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd0005040 con 0x7fafe00f5410 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.514+0000 7fafdf7fe700 1 --2- 172.21.15.149:0/4001791057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe00f5410 0x7fafe0104610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fafd0005010 tx=0x7fafd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.514+0000 7fafd7fff700 1 -- 172.21.15.149:0/4001791057 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafd0009320 con 0x7fafe00f5410 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.514+0000 7fafe5f76700 1 -- 172.21.15.149:0/4001791057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe0104d00 con 0x7fafe00f5410 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.514+0000 7fafd7fff700 1 -- 172.21.15.149:0/4001791057 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fafd0013380 con 0x7fafe00f5410 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.514+0000 7fafd7fff700 1 -- 172.21.15.149:0/4001791057 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafd0014e50 con 0x7fafe00f5410 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.514+0000 7fafe5f76700 1 -- 172.21.15.149:0/4001791057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe0105190 con 0x7fafe00f5410 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.515+0000 7fafd7fff700 1 -- 172.21.15.149:0/4001791057 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafd0014420 con 0x7fafe00f5410 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.515+0000 7fafd7fff700 1 --2- 172.21.15.149:0/4001791057 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafcc032150 0x7fafcc034610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.515+0000 7fafe5f76700 1 -- 172.21.15.149:0/4001791057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe0045ad0 con 0x7fafe00f5410 2024-02-12T16:31:15.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.515+0000 7fafd7fff700 1 -- 172.21.15.149:0/4001791057 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafd0040e30 con 0x7fafe00f5410 2024-02-12T16:31:15.519 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.515+0000 7fafdeffd700 1 --2- 172.21.15.149:0/4001791057 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafcc032150 0x7fafcc034610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:15.519 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.516+0000 7fafdeffd700 1 --2- 172.21.15.149:0/4001791057 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafcc032150 0x7fafcc034610 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fafc80096a0 tx=0x7fafc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:15.522 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.520+0000 7fafd7fff700 1 -- 172.21.15.149:0/4001791057 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafd0014790 con 0x7fafe00f5410 2024-02-12T16:31:15.726 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.724+0000 7fafe5f76700 1 -- 172.21.15.149:0/4001791057 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafe0105fe0 con 0x7fafe00f5410 2024-02-12T16:31:15.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.725+0000 7fafd7fff700 1 -- 172.21.15.149:0/4001791057 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fafd002c090 con 0x7fafe00f5410 2024-02-12T16:31:15.727 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:15.727 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:15.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.727+0000 7fafe5f76700 1 -- 172.21.15.149:0/4001791057 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafcc032150 msgr2=0x7fafcc034610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:15.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.727+0000 7fafe5f76700 1 --2- 172.21.15.149:0/4001791057 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafcc032150 0x7fafcc034610 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fafc80096a0 tx=0x7fafc8006b40).stop 2024-02-12T16:31:15.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.727+0000 7fafe5f76700 1 -- 172.21.15.149:0/4001791057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe00f5410 msgr2=0x7fafe0104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:15.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.727+0000 7fafe5f76700 1 --2- 172.21.15.149:0/4001791057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe00f5410 0x7fafe0104610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fafd0005010 tx=0x7fafd000b040).stop 2024-02-12T16:31:15.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.727+0000 7fafe5f76700 1 -- 172.21.15.149:0/4001791057 shutdown_connections 2024-02-12T16:31:15.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.727+0000 7fafe5f76700 1 --2- 172.21.15.149:0/4001791057 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafcc032150 0x7fafcc034610 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:15.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.727+0000 7fafe5f76700 1 --2- 172.21.15.149:0/4001791057 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe00f5410 0x7fafe0104610 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:15.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.727+0000 7fafe5f76700 1 -- 172.21.15.149:0/4001791057 >> 172.21.15.149:0/4001791057 conn(0x7fafe00f0b70 msgr2=0x7fafe00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:15.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.727+0000 7fafe5f76700 1 -- 172.21.15.149:0/4001791057 shutdown_connections 2024-02-12T16:31:15.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:15.727+0000 7fafe5f76700 1 -- 172.21.15.149:0/4001791057 wait complete. 2024-02-12T16:31:15.730 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:16.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:14.289395+0000 mgr.smithi104.ohwbhf (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:16.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:15.726312+0000 mon.smithi104 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.149:0/4001791057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:17.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:17.432 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:18.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:16.289621+0000 mgr.smithi104.ohwbhf (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:19.193 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.189+0000 7f35354b5700 1 -- 172.21.15.149:0/1905305788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35300f5410 msgr2=0x7f35300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:19.193 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.189+0000 7f35354b5700 1 --2- 172.21.15.149:0/1905305788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35300f5410 0x7f35300f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f3524009fb0 tx=0x7f3524009b30).stop 2024-02-12T16:31:19.193 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.189+0000 7f35354b5700 1 -- 172.21.15.149:0/1905305788 shutdown_connections 2024-02-12T16:31:19.193 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.189+0000 7f35354b5700 1 --2- 172.21.15.149:0/1905305788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35300f5410 0x7f35300f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:19.193 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.189+0000 7f35354b5700 1 -- 172.21.15.149:0/1905305788 >> 172.21.15.149:0/1905305788 conn(0x7f35300f0b70 msgr2=0x7f35300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.189+0000 7f35354b5700 1 -- 172.21.15.149:0/1905305788 shutdown_connections 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.189+0000 7f35354b5700 1 -- 172.21.15.149:0/1905305788 wait complete. 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.190+0000 7f35354b5700 1 Processor -- start 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.190+0000 7f35354b5700 1 -- start start 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.190+0000 7f35354b5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35300f5410 0x7f3530109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.190+0000 7f35354b5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3530060240 con 0x7f35300f5410 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.190+0000 7f352effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35300f5410 0x7f3530109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.190+0000 7f352effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35300f5410 0x7f3530109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:45302/0 (socket says 172.21.15.149:45302) 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.190+0000 7f352effd700 1 -- 172.21.15.149:0/792003777 learned_addr learned my addr 172.21.15.149:0/792003777 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.191+0000 7f352effd700 1 -- 172.21.15.149:0/792003777 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3524005040 con 0x7f35300f5410 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.191+0000 7f352effd700 1 --2- 172.21.15.149:0/792003777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35300f5410 0x7f3530109510 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3524005010 tx=0x7f352400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.191+0000 7f35177fe700 1 -- 172.21.15.149:0/792003777 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3524009320 con 0x7f35300f5410 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.191+0000 7f35354b5700 1 -- 172.21.15.149:0/792003777 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3530060440 con 0x7f35300f5410 2024-02-12T16:31:19.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.191+0000 7f35177fe700 1 -- 172.21.15.149:0/792003777 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3524013380 con 0x7f35300f5410 2024-02-12T16:31:19.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.191+0000 7f35177fe700 1 -- 172.21.15.149:0/792003777 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3524014e50 con 0x7f35300f5410 2024-02-12T16:31:19.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.191+0000 7f35354b5700 1 -- 172.21.15.149:0/792003777 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3530109bc0 con 0x7f35300f5410 2024-02-12T16:31:19.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.192+0000 7f35177fe700 1 -- 172.21.15.149:0/792003777 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35240095f0 con 0x7f35300f5410 2024-02-12T16:31:19.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.192+0000 7f35354b5700 1 -- 172.21.15.149:0/792003777 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3530045ad0 con 0x7f35300f5410 2024-02-12T16:31:19.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.192+0000 7f35177fe700 1 --2- 172.21.15.149:0/792003777 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f351802ff10 0x7f35180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:19.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.192+0000 7f35177fe700 1 -- 172.21.15.149:0/792003777 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3524040e70 con 0x7f35300f5410 2024-02-12T16:31:19.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.192+0000 7f352e7fc700 1 --2- 172.21.15.149:0/792003777 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f351802ff10 0x7f35180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:19.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.193+0000 7f352e7fc700 1 --2- 172.21.15.149:0/792003777 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f351802ff10 0x7f35180323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f35200096a0 tx=0x7f3520006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:19.199 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.197+0000 7f35177fe700 1 -- 172.21.15.149:0/792003777 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3524014420 con 0x7f35300f5410 2024-02-12T16:31:19.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.401+0000 7f35354b5700 1 -- 172.21.15.149:0/792003777 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f353010a3d0 con 0x7f35300f5410 2024-02-12T16:31:19.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.401+0000 7f35177fe700 1 -- 172.21.15.149:0/792003777 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3524010230 con 0x7f35300f5410 2024-02-12T16:31:19.403 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:19.404 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:19.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.403+0000 7f35354b5700 1 -- 172.21.15.149:0/792003777 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f351802ff10 msgr2=0x7f35180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:19.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.403+0000 7f35354b5700 1 --2- 172.21.15.149:0/792003777 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f351802ff10 0x7f35180323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f35200096a0 tx=0x7f3520006b40).stop 2024-02-12T16:31:19.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.403+0000 7f35354b5700 1 -- 172.21.15.149:0/792003777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35300f5410 msgr2=0x7f3530109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:19.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.403+0000 7f35354b5700 1 --2- 172.21.15.149:0/792003777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35300f5410 0x7f3530109510 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3524005010 tx=0x7f352400b040).stop 2024-02-12T16:31:19.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.403+0000 7f35354b5700 1 -- 172.21.15.149:0/792003777 shutdown_connections 2024-02-12T16:31:19.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.403+0000 7f35354b5700 1 --2- 172.21.15.149:0/792003777 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f351802ff10 0x7f35180323d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:19.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.403+0000 7f35354b5700 1 --2- 172.21.15.149:0/792003777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35300f5410 0x7f3530109510 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:19.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.403+0000 7f35354b5700 1 -- 172.21.15.149:0/792003777 >> 172.21.15.149:0/792003777 conn(0x7f35300f0b70 msgr2=0x7f35300f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:19.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.403+0000 7f35354b5700 1 -- 172.21.15.149:0/792003777 shutdown_connections 2024-02-12T16:31:19.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:19.403+0000 7f35354b5700 1 -- 172.21.15.149:0/792003777 wait complete. 2024-02-12T16:31:19.406 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:20.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:18.289891+0000 mgr.smithi104.ohwbhf (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:20.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:19.402498+0000 mon.smithi104 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.149:0/792003777' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:21.010 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:21.010 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:22.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:20.290126+0000 mgr.smithi104.ohwbhf (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:22.934 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.932+0000 7f4a40f3c700 1 -- 172.21.15.149:0/4190757974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a3c0f5410 msgr2=0x7f4a3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:22.934 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.932+0000 7f4a40f3c700 1 --2- 172.21.15.149:0/4190757974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a3c0f5410 0x7f4a3c0f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f4a30009fb0 tx=0x7f4a30009b30).stop 2024-02-12T16:31:22.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.932+0000 7f4a40f3c700 1 -- 172.21.15.149:0/4190757974 shutdown_connections 2024-02-12T16:31:22.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.932+0000 7f4a40f3c700 1 --2- 172.21.15.149:0/4190757974 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a3c0f5410 0x7f4a3c0f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:22.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.932+0000 7f4a40f3c700 1 -- 172.21.15.149:0/4190757974 >> 172.21.15.149:0/4190757974 conn(0x7f4a3c0f0b70 msgr2=0x7f4a3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:22.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.932+0000 7f4a40f3c700 1 -- 172.21.15.149:0/4190757974 shutdown_connections 2024-02-12T16:31:22.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.932+0000 7f4a40f3c700 1 -- 172.21.15.149:0/4190757974 wait complete. 2024-02-12T16:31:22.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.933+0000 7f4a40f3c700 1 Processor -- start 2024-02-12T16:31:22.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.933+0000 7f4a40f3c700 1 -- start start 2024-02-12T16:31:22.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.933+0000 7f4a40f3c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a3c0f5410 0x7f4a3c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.933+0000 7f4a40f3c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c060220 con 0x7f4a3c0f5410 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.933+0000 7f4a3a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a3c0f5410 0x7f4a3c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.933+0000 7f4a3a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a3c0f5410 0x7f4a3c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:45308/0 (socket says 172.21.15.149:45308) 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.933+0000 7f4a3a59c700 1 -- 172.21.15.149:0/339883313 learned_addr learned my addr 172.21.15.149:0/339883313 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.933+0000 7f4a3a59c700 1 -- 172.21.15.149:0/339883313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a30005040 con 0x7f4a3c0f5410 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.933+0000 7f4a3a59c700 1 --2- 172.21.15.149:0/339883313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a3c0f5410 0x7f4a3c109500 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f4a30005010 tx=0x7f4a3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.934+0000 7f4a2affd700 1 -- 172.21.15.149:0/339883313 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a30009320 con 0x7f4a3c0f5410 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.934+0000 7f4a40f3c700 1 -- 172.21.15.149:0/339883313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3c060420 con 0x7f4a3c0f5410 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.934+0000 7f4a2affd700 1 -- 172.21.15.149:0/339883313 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a30013380 con 0x7f4a3c0f5410 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.934+0000 7f4a2affd700 1 -- 172.21.15.149:0/339883313 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a30014e50 con 0x7f4a3c0f5410 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.934+0000 7f4a40f3c700 1 -- 172.21.15.149:0/339883313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3c109bb0 con 0x7f4a3c0f5410 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.934+0000 7f4a2affd700 1 -- 172.21.15.149:0/339883313 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a30014420 con 0x7f4a3c0f5410 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.934+0000 7f4a40f3c700 1 -- 172.21.15.149:0/339883313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3c059100 con 0x7f4a3c0f5410 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.935+0000 7f4a2affd700 1 --2- 172.21.15.149:0/339883313 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a2402ff10 0x7f4a240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:22.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.935+0000 7f4a2affd700 1 -- 172.21.15.149:0/339883313 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a30040f40 con 0x7f4a3c0f5410 2024-02-12T16:31:22.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.935+0000 7f4a39d9b700 1 --2- 172.21.15.149:0/339883313 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a2402ff10 0x7f4a240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:22.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.935+0000 7f4a39d9b700 1 --2- 172.21.15.149:0/339883313 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a2402ff10 0x7f4a240323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f4a2c0096a0 tx=0x7f4a2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:22.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:22.939+0000 7f4a2affd700 1 -- 172.21.15.149:0/339883313 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a300095f0 con 0x7f4a3c0f5410 2024-02-12T16:31:23.146 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.144+0000 7f4a40f3c700 1 -- 172.21.15.149:0/339883313 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a3c045ad0 con 0x7f4a3c0f5410 2024-02-12T16:31:23.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.145+0000 7f4a2affd700 1 -- 172.21.15.149:0/339883313 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4a3001db70 con 0x7f4a3c0f5410 2024-02-12T16:31:23.147 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:23.147 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:23.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.147+0000 7f4a40f3c700 1 -- 172.21.15.149:0/339883313 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a2402ff10 msgr2=0x7f4a240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:23.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.147+0000 7f4a40f3c700 1 --2- 172.21.15.149:0/339883313 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a2402ff10 0x7f4a240323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f4a2c0096a0 tx=0x7f4a2c006b40).stop 2024-02-12T16:31:23.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.147+0000 7f4a40f3c700 1 -- 172.21.15.149:0/339883313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a3c0f5410 msgr2=0x7f4a3c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:23.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.147+0000 7f4a40f3c700 1 --2- 172.21.15.149:0/339883313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a3c0f5410 0x7f4a3c109500 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f4a30005010 tx=0x7f4a3000b040).stop 2024-02-12T16:31:23.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.147+0000 7f4a40f3c700 1 -- 172.21.15.149:0/339883313 shutdown_connections 2024-02-12T16:31:23.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.147+0000 7f4a40f3c700 1 --2- 172.21.15.149:0/339883313 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a2402ff10 0x7f4a240323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:23.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.147+0000 7f4a40f3c700 1 --2- 172.21.15.149:0/339883313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a3c0f5410 0x7f4a3c109500 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:23.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.147+0000 7f4a40f3c700 1 -- 172.21.15.149:0/339883313 >> 172.21.15.149:0/339883313 conn(0x7f4a3c0f0b70 msgr2=0x7f4a3c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:23.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.147+0000 7f4a40f3c700 1 -- 172.21.15.149:0/339883313 shutdown_connections 2024-02-12T16:31:23.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:23.147+0000 7f4a40f3c700 1 -- 172.21.15.149:0/339883313 wait complete. 2024-02-12T16:31:23.150 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:24.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:22.290440+0000 mgr.smithi104.ohwbhf (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:24.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:23.146173+0000 mon.smithi104 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.149:0/339883313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:24.775 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:24.775 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:24.403274+0000 mon.smithi104 (mon.0) 141 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:31:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:24.410526+0000 mon.smithi104 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:31:26.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:24.290745+0000 mgr.smithi104.ohwbhf (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:26.527 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.523+0000 7fa1bb745700 1 -- 172.21.15.149:0/3583548299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1b40f5410 msgr2=0x7fa1b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:26.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.523+0000 7fa1bb745700 1 --2- 172.21.15.149:0/3583548299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1b40f5410 0x7fa1b40f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa1a4009fb0 tx=0x7fa1a4009b30).stop 2024-02-12T16:31:26.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.523+0000 7fa1bb745700 1 -- 172.21.15.149:0/3583548299 shutdown_connections 2024-02-12T16:31:26.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.523+0000 7fa1bb745700 1 --2- 172.21.15.149:0/3583548299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1b40f5410 0x7fa1b40f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:26.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.523+0000 7fa1bb745700 1 -- 172.21.15.149:0/3583548299 >> 172.21.15.149:0/3583548299 conn(0x7fa1b40f0b70 msgr2=0x7fa1b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:26.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.523+0000 7fa1bb745700 1 -- 172.21.15.149:0/3583548299 shutdown_connections 2024-02-12T16:31:26.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.523+0000 7fa1bb745700 1 -- 172.21.15.149:0/3583548299 wait complete. 2024-02-12T16:31:26.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.524+0000 7fa1bb745700 1 Processor -- start 2024-02-12T16:31:26.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.524+0000 7fa1bb745700 1 -- start start 2024-02-12T16:31:26.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.524+0000 7fa1bb745700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1b40f5410 0x7fa1b4109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:26.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.524+0000 7fa1bb745700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b4060220 con 0x7fa1b40f5410 2024-02-12T16:31:26.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.524+0000 7fa1b94e1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1b40f5410 0x7fa1b4109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:26.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.524+0000 7fa1b94e1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1b40f5410 0x7fa1b4109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:60526/0 (socket says 172.21.15.149:60526) 2024-02-12T16:31:26.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.524+0000 7fa1b94e1700 1 -- 172.21.15.149:0/4247020680 learned_addr learned my addr 172.21.15.149:0/4247020680 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:26.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.525+0000 7fa1b94e1700 1 -- 172.21.15.149:0/4247020680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a4005040 con 0x7fa1b40f5410 2024-02-12T16:31:26.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.525+0000 7fa1b94e1700 1 --2- 172.21.15.149:0/4247020680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1b40f5410 0x7fa1b4109500 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa1a4005010 tx=0x7fa1a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:26.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.525+0000 7fa1a9ffb700 1 -- 172.21.15.149:0/4247020680 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1a4009320 con 0x7fa1b40f5410 2024-02-12T16:31:26.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.525+0000 7fa1bb745700 1 -- 172.21.15.149:0/4247020680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b4060420 con 0x7fa1b40f5410 2024-02-12T16:31:26.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.525+0000 7fa1a9ffb700 1 -- 172.21.15.149:0/4247020680 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1a4013380 con 0x7fa1b40f5410 2024-02-12T16:31:26.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.525+0000 7fa1a9ffb700 1 -- 172.21.15.149:0/4247020680 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1a4014e50 con 0x7fa1b40f5410 2024-02-12T16:31:26.529 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.525+0000 7fa1bb745700 1 -- 172.21.15.149:0/4247020680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b4109bb0 con 0x7fa1b40f5410 2024-02-12T16:31:26.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.526+0000 7fa1a9ffb700 1 -- 172.21.15.149:0/4247020680 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1a40095f0 con 0x7fa1b40f5410 2024-02-12T16:31:26.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.526+0000 7fa1bb745700 1 -- 172.21.15.149:0/4247020680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b4059100 con 0x7fa1b40f5410 2024-02-12T16:31:26.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.526+0000 7fa1a9ffb700 1 --2- 172.21.15.149:0/4247020680 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa1a002ff10 0x7fa1a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:26.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.526+0000 7fa1a9ffb700 1 -- 172.21.15.149:0/4247020680 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1a4040e90 con 0x7fa1b40f5410 2024-02-12T16:31:26.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.526+0000 7fa1b8ce0700 1 --2- 172.21.15.149:0/4247020680 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa1a002ff10 0x7fa1a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:26.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.527+0000 7fa1b8ce0700 1 --2- 172.21.15.149:0/4247020680 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa1a002ff10 0x7fa1a00323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa1b00096a0 tx=0x7fa1b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:26.533 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.531+0000 7fa1a9ffb700 1 -- 172.21.15.149:0/4247020680 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1a4014420 con 0x7fa1b40f5410 2024-02-12T16:31:26.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.735+0000 7fa1bb745700 1 -- 172.21.15.149:0/4247020680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1b4045ad0 con 0x7fa1b40f5410 2024-02-12T16:31:26.738 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.736+0000 7fa1a9ffb700 1 -- 172.21.15.149:0/4247020680 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa1a4010230 con 0x7fa1b40f5410 2024-02-12T16:31:26.738 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:26.738 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:26.740 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.737+0000 7fa1bb745700 1 -- 172.21.15.149:0/4247020680 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa1a002ff10 msgr2=0x7fa1a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:26.740 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.737+0000 7fa1bb745700 1 --2- 172.21.15.149:0/4247020680 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa1a002ff10 0x7fa1a00323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa1b00096a0 tx=0x7fa1b0006b40).stop 2024-02-12T16:31:26.741 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.737+0000 7fa1bb745700 1 -- 172.21.15.149:0/4247020680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1b40f5410 msgr2=0x7fa1b4109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:26.741 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.737+0000 7fa1bb745700 1 --2- 172.21.15.149:0/4247020680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1b40f5410 0x7fa1b4109500 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa1a4005010 tx=0x7fa1a400b040).stop 2024-02-12T16:31:26.741 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.738+0000 7fa1bb745700 1 -- 172.21.15.149:0/4247020680 shutdown_connections 2024-02-12T16:31:26.741 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.738+0000 7fa1bb745700 1 --2- 172.21.15.149:0/4247020680 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa1a002ff10 0x7fa1a00323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:26.741 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.738+0000 7fa1bb745700 1 --2- 172.21.15.149:0/4247020680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1b40f5410 0x7fa1b4109500 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:26.741 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.738+0000 7fa1bb745700 1 -- 172.21.15.149:0/4247020680 >> 172.21.15.149:0/4247020680 conn(0x7fa1b40f0b70 msgr2=0x7fa1b40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:26.741 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.738+0000 7fa1bb745700 1 -- 172.21.15.149:0/4247020680 shutdown_connections 2024-02-12T16:31:26.741 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:26.738+0000 7fa1bb745700 1 -- 172.21.15.149:0/4247020680 wait complete. 2024-02-12T16:31:26.741 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:27.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:26.737151+0000 mon.smithi104 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.149:0/4247020680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:28.377 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:28.377 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:28.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:26.291031+0000 mgr.smithi104.ohwbhf (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:30.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.133+0000 7fcd710e2700 1 -- 172.21.15.149:0/1775846118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd6c0f5410 msgr2=0x7fcd6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:30.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.133+0000 7fcd710e2700 1 --2- 172.21.15.149:0/1775846118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd6c0f5410 0x7fcd6c0f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fcd54009fb0 tx=0x7fcd54009b30).stop 2024-02-12T16:31:30.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.133+0000 7fcd710e2700 1 -- 172.21.15.149:0/1775846118 shutdown_connections 2024-02-12T16:31:30.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.133+0000 7fcd710e2700 1 --2- 172.21.15.149:0/1775846118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd6c0f5410 0x7fcd6c0f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:30.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.133+0000 7fcd710e2700 1 -- 172.21.15.149:0/1775846118 >> 172.21.15.149:0/1775846118 conn(0x7fcd6c0f0b70 msgr2=0x7fcd6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:30.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.133+0000 7fcd710e2700 1 -- 172.21.15.149:0/1775846118 shutdown_connections 2024-02-12T16:31:30.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.133+0000 7fcd710e2700 1 -- 172.21.15.149:0/1775846118 wait complete. 2024-02-12T16:31:30.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.134+0000 7fcd710e2700 1 Processor -- start 2024-02-12T16:31:30.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.134+0000 7fcd710e2700 1 -- start start 2024-02-12T16:31:30.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.134+0000 7fcd710e2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd6c0f5410 0x7fcd6c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:30.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.134+0000 7fcd710e2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c0fb960 con 0x7fcd6c0f5410 2024-02-12T16:31:30.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.134+0000 7fcd6ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd6c0f5410 0x7fcd6c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:30.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.134+0000 7fcd6ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd6c0f5410 0x7fcd6c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:60542/0 (socket says 172.21.15.149:60542) 2024-02-12T16:31:30.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.134+0000 7fcd6ad9d700 1 -- 172.21.15.149:0/1393670398 learned_addr learned my addr 172.21.15.149:0/1393670398 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:30.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.134+0000 7fcd6ad9d700 1 -- 172.21.15.149:0/1393670398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd54005040 con 0x7fcd6c0f5410 2024-02-12T16:31:30.138 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.134+0000 7fcd6ad9d700 1 --2- 172.21.15.149:0/1393670398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd6c0f5410 0x7fcd6c0fd4e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fcd54005010 tx=0x7fcd5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:30.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.135+0000 7fcd637fe700 1 -- 172.21.15.149:0/1393670398 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd54009320 con 0x7fcd6c0f5410 2024-02-12T16:31:30.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.135+0000 7fcd710e2700 1 -- 172.21.15.149:0/1393670398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd6c0fbb00 con 0x7fcd6c0f5410 2024-02-12T16:31:30.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.135+0000 7fcd637fe700 1 -- 172.21.15.149:0/1393670398 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd54013380 con 0x7fcd6c0f5410 2024-02-12T16:31:30.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.135+0000 7fcd637fe700 1 -- 172.21.15.149:0/1393670398 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd54014e50 con 0x7fcd6c0f5410 2024-02-12T16:31:30.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.135+0000 7fcd710e2700 1 -- 172.21.15.149:0/1393670398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd6c0fbf90 con 0x7fcd6c0f5410 2024-02-12T16:31:30.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.136+0000 7fcd637fe700 1 -- 172.21.15.149:0/1393670398 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd54014420 con 0x7fcd6c0f5410 2024-02-12T16:31:30.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.136+0000 7fcd710e2700 1 -- 172.21.15.149:0/1393670398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6c059100 con 0x7fcd6c0f5410 2024-02-12T16:31:30.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.136+0000 7fcd637fe700 1 --2- 172.21.15.149:0/1393670398 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd5802ff10 0x7fcd580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:30.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.136+0000 7fcd637fe700 1 -- 172.21.15.149:0/1393670398 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd54040e30 con 0x7fcd6c0f5410 2024-02-12T16:31:30.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.136+0000 7fcd6a59c700 1 --2- 172.21.15.149:0/1393670398 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd5802ff10 0x7fcd580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:30.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.136+0000 7fcd6a59c700 1 --2- 172.21.15.149:0/1393670398 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd5802ff10 0x7fcd580323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fcd5c0096a0 tx=0x7fcd5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:30.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.140+0000 7fcd637fe700 1 -- 172.21.15.149:0/1393670398 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd540095f0 con 0x7fcd6c0f5410 2024-02-12T16:31:30.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.344+0000 7fcd710e2700 1 -- 172.21.15.149:0/1393670398 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd6c0fafa0 con 0x7fcd6c0f5410 2024-02-12T16:31:30.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.344+0000 7fcd637fe700 1 -- 172.21.15.149:0/1393670398 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd5401db70 con 0x7fcd6c0f5410 2024-02-12T16:31:30.347 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:30.347 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:30.348 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.346+0000 7fcd710e2700 1 -- 172.21.15.149:0/1393670398 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd5802ff10 msgr2=0x7fcd580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:30.349 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.346+0000 7fcd710e2700 1 --2- 172.21.15.149:0/1393670398 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd5802ff10 0x7fcd580323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fcd5c0096a0 tx=0x7fcd5c006b40).stop 2024-02-12T16:31:30.349 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.346+0000 7fcd710e2700 1 -- 172.21.15.149:0/1393670398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd6c0f5410 msgr2=0x7fcd6c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:30.349 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.346+0000 7fcd710e2700 1 --2- 172.21.15.149:0/1393670398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd6c0f5410 0x7fcd6c0fd4e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fcd54005010 tx=0x7fcd5400b040).stop 2024-02-12T16:31:30.349 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.346+0000 7fcd710e2700 1 -- 172.21.15.149:0/1393670398 shutdown_connections 2024-02-12T16:31:30.349 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.346+0000 7fcd710e2700 1 --2- 172.21.15.149:0/1393670398 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd5802ff10 0x7fcd580323d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:30.349 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.346+0000 7fcd710e2700 1 --2- 172.21.15.149:0/1393670398 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd6c0f5410 0x7fcd6c0fd4e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:30.349 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.346+0000 7fcd710e2700 1 -- 172.21.15.149:0/1393670398 >> 172.21.15.149:0/1393670398 conn(0x7fcd6c0f0b70 msgr2=0x7fcd6c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:30.349 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.346+0000 7fcd710e2700 1 -- 172.21.15.149:0/1393670398 shutdown_connections 2024-02-12T16:31:30.349 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:30.346+0000 7fcd710e2700 1 -- 172.21.15.149:0/1393670398 wait complete. 2024-02-12T16:31:30.349 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:30.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:28.291262+0000 mgr.smithi104.ohwbhf (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:31.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:31 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:30.345749+0000 mon.smithi104 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.149:0/1393670398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:32.016 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:32.017 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:32.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:30.291580+0000 mgr.smithi104.ohwbhf (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:33.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.660+0000 7f8042bb4700 1 -- 172.21.15.149:0/892175069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c060220 msgr2=0x7f803c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:33.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.660+0000 7f8042bb4700 1 --2- 172.21.15.149:0/892175069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c060220 0x7f803c0f9150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8030009fb0 tx=0x7f8030009b30).stop 2024-02-12T16:31:33.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.660+0000 7f8042bb4700 1 -- 172.21.15.149:0/892175069 shutdown_connections 2024-02-12T16:31:33.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.660+0000 7f8042bb4700 1 --2- 172.21.15.149:0/892175069 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c060220 0x7f803c0f9150 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:33.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.660+0000 7f8042bb4700 1 -- 172.21.15.149:0/892175069 >> 172.21.15.149:0/892175069 conn(0x7f803c0f0b70 msgr2=0x7f803c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:33.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.660+0000 7f8042bb4700 1 -- 172.21.15.149:0/892175069 shutdown_connections 2024-02-12T16:31:33.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.660+0000 7f8042bb4700 1 -- 172.21.15.149:0/892175069 wait complete. 2024-02-12T16:31:33.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.661+0000 7f8042bb4700 1 Processor -- start 2024-02-12T16:31:33.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.661+0000 7f8042bb4700 1 -- start start 2024-02-12T16:31:33.665 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.661+0000 7f8042bb4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c060220 0x7f803c1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:33.665 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.661+0000 7f8042bb4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803c106c90 con 0x7f803c060220 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.661+0000 7f8040950700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c060220 0x7f803c1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.661+0000 7f8040950700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c060220 0x7f803c1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:60550/0 (socket says 172.21.15.149:60550) 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.661+0000 7f8040950700 1 -- 172.21.15.149:0/1381372758 learned_addr learned my addr 172.21.15.149:0/1381372758 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.662+0000 7f8040950700 1 -- 172.21.15.149:0/1381372758 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8030005040 con 0x7f803c060220 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.662+0000 7f8040950700 1 --2- 172.21.15.149:0/1381372758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c060220 0x7f803c1067a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f80300089f0 tx=0x7f803000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.662+0000 7f80397fa700 1 -- 172.21.15.149:0/1381372758 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8030009320 con 0x7f803c060220 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.662+0000 7f80397fa700 1 -- 172.21.15.149:0/1381372758 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8030013380 con 0x7f803c060220 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.662+0000 7f8042bb4700 1 -- 172.21.15.149:0/1381372758 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f803c106e90 con 0x7f803c060220 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.662+0000 7f80397fa700 1 -- 172.21.15.149:0/1381372758 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8030014e20 con 0x7f803c060220 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.662+0000 7f8042bb4700 1 -- 172.21.15.149:0/1381372758 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803c107320 con 0x7f803c060220 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.663+0000 7f80397fa700 1 -- 172.21.15.149:0/1381372758 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8030014420 con 0x7f803c060220 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.663+0000 7f8042bb4700 1 -- 172.21.15.149:0/1381372758 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803c045ad0 con 0x7f803c060220 2024-02-12T16:31:33.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.663+0000 7f80397fa700 1 --2- 172.21.15.149:0/1381372758 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f802c02ff10 0x7f802c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:33.667 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.663+0000 7f80397fa700 1 -- 172.21.15.149:0/1381372758 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8030040f50 con 0x7f803c060220 2024-02-12T16:31:33.667 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.663+0000 7f803bfff700 1 --2- 172.21.15.149:0/1381372758 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f802c02ff10 0x7f802c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:33.667 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.663+0000 7f803bfff700 1 --2- 172.21.15.149:0/1381372758 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f802c02ff10 0x7f802c0323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f80280096a0 tx=0x7f8028006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:33.669 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.668+0000 7f80397fa700 1 -- 172.21.15.149:0/1381372758 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f803006d2b0 con 0x7f803c060220 2024-02-12T16:31:33.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.873+0000 7f8042bb4700 1 -- 172.21.15.149:0/1381372758 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f803c1081d0 con 0x7f803c060220 2024-02-12T16:31:33.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.873+0000 7f80397fa700 1 -- 172.21.15.149:0/1381372758 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f80300095f0 con 0x7f803c060220 2024-02-12T16:31:33.876 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:33.876 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:33.877 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.875+0000 7f8042bb4700 1 -- 172.21.15.149:0/1381372758 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f802c02ff10 msgr2=0x7f802c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:33.878 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.875+0000 7f8042bb4700 1 --2- 172.21.15.149:0/1381372758 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f802c02ff10 0x7f802c0323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f80280096a0 tx=0x7f8028006b40).stop 2024-02-12T16:31:33.878 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.875+0000 7f8042bb4700 1 -- 172.21.15.149:0/1381372758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c060220 msgr2=0x7f803c1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:33.878 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.875+0000 7f8042bb4700 1 --2- 172.21.15.149:0/1381372758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c060220 0x7f803c1067a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f80300089f0 tx=0x7f803000b040).stop 2024-02-12T16:31:33.878 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.875+0000 7f8042bb4700 1 -- 172.21.15.149:0/1381372758 shutdown_connections 2024-02-12T16:31:33.878 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.875+0000 7f8042bb4700 1 --2- 172.21.15.149:0/1381372758 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f802c02ff10 0x7f802c0323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:33.878 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.875+0000 7f8042bb4700 1 --2- 172.21.15.149:0/1381372758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f803c060220 0x7f803c1067a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:33.878 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.875+0000 7f8042bb4700 1 -- 172.21.15.149:0/1381372758 >> 172.21.15.149:0/1381372758 conn(0x7f803c0f0b70 msgr2=0x7f803c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:33.878 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.875+0000 7f8042bb4700 1 -- 172.21.15.149:0/1381372758 shutdown_connections 2024-02-12T16:31:33.878 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:33.875+0000 7f8042bb4700 1 -- 172.21.15.149:0/1381372758 wait complete. 2024-02-12T16:31:33.879 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:34.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:32.291916+0000 mgr.smithi104.ohwbhf (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:34.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:33.874827+0000 mon.smithi104 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.149:0/1381372758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:35.508 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:35.509 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:36.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:34.292218+0000 mgr.smithi104.ohwbhf (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:37.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.301+0000 7fd982b29700 1 -- 172.21.15.149:0/131948902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd97c0f51a0 msgr2=0x7fd97c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:37.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.301+0000 7fd982b29700 1 --2- 172.21.15.149:0/131948902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd97c0f51a0 0x7fd97c0f55c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fd96c009fb0 tx=0x7fd96c009b30).stop 2024-02-12T16:31:37.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.301+0000 7fd982b29700 1 -- 172.21.15.149:0/131948902 shutdown_connections 2024-02-12T16:31:37.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.301+0000 7fd982b29700 1 --2- 172.21.15.149:0/131948902 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd97c0f51a0 0x7fd97c0f55c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:37.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.301+0000 7fd982b29700 1 -- 172.21.15.149:0/131948902 >> 172.21.15.149:0/131948902 conn(0x7fd97c0f09a0 msgr2=0x7fd97c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:37.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.302+0000 7fd982b29700 1 -- 172.21.15.149:0/131948902 shutdown_connections 2024-02-12T16:31:37.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.302+0000 7fd982b29700 1 -- 172.21.15.149:0/131948902 wait complete. 2024-02-12T16:31:37.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.302+0000 7fd982b29700 1 Processor -- start 2024-02-12T16:31:37.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.302+0000 7fd982b29700 1 -- start start 2024-02-12T16:31:37.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.302+0000 7fd982b29700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd97c0fd230 0x7fd97c0fb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.302+0000 7fd982b29700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c0fd650 con 0x7fd97c0fd230 2024-02-12T16:31:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.302+0000 7fd981b27700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd97c0fd230 0x7fd97c0fb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.302+0000 7fd981b27700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd97c0fd230 0x7fd97c0fb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:60148/0 (socket says 172.21.15.149:60148) 2024-02-12T16:31:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.302+0000 7fd981b27700 1 -- 172.21.15.149:0/209981425 learned_addr learned my addr 172.21.15.149:0/209981425 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.303+0000 7fd981b27700 1 -- 172.21.15.149:0/209981425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c005040 con 0x7fd97c0fd230 2024-02-12T16:31:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.303+0000 7fd981b27700 1 --2- 172.21.15.149:0/209981425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd97c0fd230 0x7fd97c0fb6f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd96c011000 tx=0x7fd96c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.303+0000 7fd9727fc700 1 -- 172.21.15.149:0/209981425 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd96c013380 con 0x7fd97c0fd230 2024-02-12T16:31:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.303+0000 7fd9727fc700 1 -- 172.21.15.149:0/209981425 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd96c013790 con 0x7fd97c0fd230 2024-02-12T16:31:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.303+0000 7fd982b29700 1 -- 172.21.15.149:0/209981425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c0fbbe0 con 0x7fd97c0fd230 2024-02-12T16:31:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.303+0000 7fd9727fc700 1 -- 172.21.15.149:0/209981425 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd96c01ee70 con 0x7fd97c0fd230 2024-02-12T16:31:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.303+0000 7fd982b29700 1 -- 172.21.15.149:0/209981425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c0fc0d0 con 0x7fd97c0fd230 2024-02-12T16:31:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.304+0000 7fd9727fc700 1 -- 172.21.15.149:0/209981425 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd96c025060 con 0x7fd97c0fd230 2024-02-12T16:31:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.304+0000 7fd982b29700 1 -- 172.21.15.149:0/209981425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97c045ad0 con 0x7fd97c0fd230 2024-02-12T16:31:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.304+0000 7fd9727fc700 1 --2- 172.21.15.149:0/209981425 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd96802ff60 0x7fd968032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.304+0000 7fd9727fc700 1 -- 172.21.15.149:0/209981425 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd96c040e50 con 0x7fd97c0fd230 2024-02-12T16:31:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.304+0000 7fd981326700 1 --2- 172.21.15.149:0/209981425 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd96802ff60 0x7fd968032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.305+0000 7fd981326700 1 --2- 172.21.15.149:0/209981425 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd96802ff60 0x7fd968032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd9780096a0 tx=0x7fd978006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:37.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.309+0000 7fd9727fc700 1 -- 172.21.15.149:0/209981425 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd96c01c950 con 0x7fd97c0fd230 2024-02-12T16:31:37.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.513+0000 7fd982b29700 1 -- 172.21.15.149:0/209981425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd97c10c8a0 con 0x7fd97c0fd230 2024-02-12T16:31:37.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.513+0000 7fd9727fc700 1 -- 172.21.15.149:0/209981425 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd96c02d2d0 con 0x7fd97c0fd230 2024-02-12T16:31:37.516 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:37.516 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:37.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.515+0000 7fd982b29700 1 -- 172.21.15.149:0/209981425 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd96802ff60 msgr2=0x7fd968032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:37.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.515+0000 7fd982b29700 1 --2- 172.21.15.149:0/209981425 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd96802ff60 0x7fd968032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd9780096a0 tx=0x7fd978006b40).stop 2024-02-12T16:31:37.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.515+0000 7fd982b29700 1 -- 172.21.15.149:0/209981425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd97c0fd230 msgr2=0x7fd97c0fb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:37.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.515+0000 7fd982b29700 1 --2- 172.21.15.149:0/209981425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd97c0fd230 0x7fd97c0fb6f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd96c011000 tx=0x7fd96c009b30).stop 2024-02-12T16:31:37.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.515+0000 7fd982b29700 1 -- 172.21.15.149:0/209981425 shutdown_connections 2024-02-12T16:31:37.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.515+0000 7fd982b29700 1 --2- 172.21.15.149:0/209981425 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd96802ff60 0x7fd968032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:37.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.515+0000 7fd982b29700 1 --2- 172.21.15.149:0/209981425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd97c0fd230 0x7fd97c0fb6f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:37.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.515+0000 7fd982b29700 1 -- 172.21.15.149:0/209981425 >> 172.21.15.149:0/209981425 conn(0x7fd97c0f09a0 msgr2=0x7fd97c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:37.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.515+0000 7fd982b29700 1 -- 172.21.15.149:0/209981425 shutdown_connections 2024-02-12T16:31:37.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:37.515+0000 7fd982b29700 1 -- 172.21.15.149:0/209981425 wait complete. 2024-02-12T16:31:37.518 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:38.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:36.292453+0000 mgr.smithi104.ohwbhf (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:38.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:37.514724+0000 mon.smithi104 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.149:0/209981425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:39.173 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:39.173 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:40.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:38.292712+0000 mgr.smithi104.ohwbhf (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:40.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.931+0000 7f5d3b34d700 1 -- 172.21.15.149:0/2849656860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d340f5410 msgr2=0x7f5d340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:40.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.931+0000 7f5d3b34d700 1 --2- 172.21.15.149:0/2849656860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d340f5410 0x7f5d340f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f5d30009fb0 tx=0x7f5d30009b30).stop 2024-02-12T16:31:40.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.931+0000 7f5d3b34d700 1 -- 172.21.15.149:0/2849656860 shutdown_connections 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.931+0000 7f5d3b34d700 1 --2- 172.21.15.149:0/2849656860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d340f5410 0x7f5d340f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.931+0000 7f5d3b34d700 1 -- 172.21.15.149:0/2849656860 >> 172.21.15.149:0/2849656860 conn(0x7f5d340f0b70 msgr2=0x7f5d340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.931+0000 7f5d3b34d700 1 -- 172.21.15.149:0/2849656860 shutdown_connections 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.931+0000 7f5d3b34d700 1 -- 172.21.15.149:0/2849656860 wait complete. 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.932+0000 7f5d3b34d700 1 Processor -- start 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.932+0000 7f5d3b34d700 1 -- start start 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.932+0000 7f5d3b34d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d340f5410 0x7f5d341067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.932+0000 7f5d3b34d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d34106cd0 con 0x7f5d340f5410 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.932+0000 7f5d390e9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d340f5410 0x7f5d341067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.932+0000 7f5d390e9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d340f5410 0x7f5d341067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:60160/0 (socket says 172.21.15.149:60160) 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.932+0000 7f5d390e9700 1 -- 172.21.15.149:0/1242900673 learned_addr learned my addr 172.21.15.149:0/1242900673 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:40.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.932+0000 7f5d390e9700 1 -- 172.21.15.149:0/1242900673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d30005040 con 0x7f5d340f5410 2024-02-12T16:31:40.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.932+0000 7f5d390e9700 1 --2- 172.21.15.149:0/1242900673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d340f5410 0x7f5d341067e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f5d300089f0 tx=0x7f5d3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:40.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.933+0000 7f5d29ffb700 1 -- 172.21.15.149:0/1242900673 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d30009320 con 0x7f5d340f5410 2024-02-12T16:31:40.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.933+0000 7f5d29ffb700 1 -- 172.21.15.149:0/1242900673 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d30013380 con 0x7f5d340f5410 2024-02-12T16:31:40.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.933+0000 7f5d29ffb700 1 -- 172.21.15.149:0/1242900673 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d3001ed80 con 0x7f5d340f5410 2024-02-12T16:31:40.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.933+0000 7f5d3b34d700 1 -- 172.21.15.149:0/1242900673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d34106ed0 con 0x7f5d340f5410 2024-02-12T16:31:40.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.933+0000 7f5d3b34d700 1 -- 172.21.15.149:0/1242900673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d34107360 con 0x7f5d340f5410 2024-02-12T16:31:40.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.934+0000 7f5d29ffb700 1 -- 172.21.15.149:0/1242900673 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d3001e420 con 0x7f5d340f5410 2024-02-12T16:31:40.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.934+0000 7f5d29ffb700 1 --2- 172.21.15.149:0/1242900673 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d2002ff60 0x7f5d20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:40.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.934+0000 7f5d3b34d700 1 -- 172.21.15.149:0/1242900673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d34045ad0 con 0x7f5d340f5410 2024-02-12T16:31:40.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.934+0000 7f5d29ffb700 1 -- 172.21.15.149:0/1242900673 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d30040f00 con 0x7f5d340f5410 2024-02-12T16:31:40.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.934+0000 7f5d388e8700 1 --2- 172.21.15.149:0/1242900673 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d2002ff60 0x7f5d20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:40.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.934+0000 7f5d388e8700 1 --2- 172.21.15.149:0/1242900673 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d2002ff60 0x7f5d20032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f5d240096a0 tx=0x7f5d24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:40.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:40.938+0000 7f5d29ffb700 1 -- 172.21.15.149:0/1242900673 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d300095f0 con 0x7f5d340f5410 2024-02-12T16:31:41.143 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.141+0000 7f5d3b34d700 1 -- 172.21.15.149:0/1242900673 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d340fb090 con 0x7f5d340f5410 2024-02-12T16:31:41.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.143+0000 7f5d29ffb700 1 -- 172.21.15.149:0/1242900673 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5d3001cb10 con 0x7f5d340f5410 2024-02-12T16:31:41.145 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:41.145 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:41.146 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.144+0000 7f5d3b34d700 1 -- 172.21.15.149:0/1242900673 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d2002ff60 msgr2=0x7f5d20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:41.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.144+0000 7f5d3b34d700 1 --2- 172.21.15.149:0/1242900673 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d2002ff60 0x7f5d20032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f5d240096a0 tx=0x7f5d24006b40).stop 2024-02-12T16:31:41.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.144+0000 7f5d3b34d700 1 -- 172.21.15.149:0/1242900673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d340f5410 msgr2=0x7f5d341067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:41.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.144+0000 7f5d3b34d700 1 --2- 172.21.15.149:0/1242900673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d340f5410 0x7f5d341067e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f5d300089f0 tx=0x7f5d3000b040).stop 2024-02-12T16:31:41.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.144+0000 7f5d3b34d700 1 -- 172.21.15.149:0/1242900673 shutdown_connections 2024-02-12T16:31:41.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.144+0000 7f5d3b34d700 1 --2- 172.21.15.149:0/1242900673 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d2002ff60 0x7f5d20032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:41.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.144+0000 7f5d3b34d700 1 --2- 172.21.15.149:0/1242900673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d340f5410 0x7f5d341067e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:41.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.144+0000 7f5d3b34d700 1 -- 172.21.15.149:0/1242900673 >> 172.21.15.149:0/1242900673 conn(0x7f5d340f0b70 msgr2=0x7f5d3405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:41.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.144+0000 7f5d3b34d700 1 -- 172.21.15.149:0/1242900673 shutdown_connections 2024-02-12T16:31:41.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:41.144+0000 7f5d3b34d700 1 -- 172.21.15.149:0/1242900673 wait complete. 2024-02-12T16:31:41.148 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:42.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:40.293026+0000 mgr.smithi104.ohwbhf (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:42.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:41.143909+0000 mon.smithi104 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.149:0/1242900673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:42.851 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:42.852 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:44.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:42.293375+0000 mgr.smithi104.ohwbhf (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:44.711 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.707+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/507777270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99c060280 msgr2=0x7fb99c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:44.711 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.707+0000 7fb9a0a8f700 1 --2- 172.21.15.149:0/507777270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99c060280 0x7fb99c0f90f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb990009fb0 tx=0x7fb990009b30).stop 2024-02-12T16:31:44.711 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.707+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/507777270 shutdown_connections 2024-02-12T16:31:44.711 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.707+0000 7fb9a0a8f700 1 --2- 172.21.15.149:0/507777270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99c060280 0x7fb99c0f90f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:44.711 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.707+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/507777270 >> 172.21.15.149:0/507777270 conn(0x7fb99c0f0b70 msgr2=0x7fb99c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:44.711 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.707+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/507777270 shutdown_connections 2024-02-12T16:31:44.711 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.707+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/507777270 wait complete. 2024-02-12T16:31:44.711 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.707+0000 7fb9a0a8f700 1 Processor -- start 2024-02-12T16:31:44.711 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.707+0000 7fb9a0a8f700 1 -- start start 2024-02-12T16:31:44.711 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.708+0000 7fb9a0a8f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99c060280 0x7fb99c106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:44.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.708+0000 7fb9a0a8f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c106c50 con 0x7fb99c060280 2024-02-12T16:31:44.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.708+0000 7fb99a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99c060280 0x7fb99c106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:44.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.708+0000 7fb99a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99c060280 0x7fb99c106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33382/0 (socket says 172.21.15.149:33382) 2024-02-12T16:31:44.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.708+0000 7fb99a59c700 1 -- 172.21.15.149:0/360846940 learned_addr learned my addr 172.21.15.149:0/360846940 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:44.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.708+0000 7fb99a59c700 1 -- 172.21.15.149:0/360846940 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb990005040 con 0x7fb99c060280 2024-02-12T16:31:44.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.708+0000 7fb99a59c700 1 --2- 172.21.15.149:0/360846940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99c060280 0x7fb99c106760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb9900089f0 tx=0x7fb99000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:44.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.708+0000 7fb98affd700 1 -- 172.21.15.149:0/360846940 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb990009320 con 0x7fb99c060280 2024-02-12T16:31:44.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.708+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/360846940 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c106e50 con 0x7fb99c060280 2024-02-12T16:31:44.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.708+0000 7fb98affd700 1 -- 172.21.15.149:0/360846940 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb990013380 con 0x7fb99c060280 2024-02-12T16:31:44.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.709+0000 7fb98affd700 1 -- 172.21.15.149:0/360846940 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb990014e20 con 0x7fb99c060280 2024-02-12T16:31:44.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.709+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/360846940 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c1072e0 con 0x7fb99c060280 2024-02-12T16:31:44.713 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.709+0000 7fb98affd700 1 -- 172.21.15.149:0/360846940 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb990014420 con 0x7fb99c060280 2024-02-12T16:31:44.713 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.709+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/360846940 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c045ad0 con 0x7fb99c060280 2024-02-12T16:31:44.713 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.709+0000 7fb98affd700 1 --2- 172.21.15.149:0/360846940 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb98402ff10 0x7fb9840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:44.713 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.709+0000 7fb98affd700 1 -- 172.21.15.149:0/360846940 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb990040f20 con 0x7fb99c060280 2024-02-12T16:31:44.713 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.710+0000 7fb999d9b700 1 --2- 172.21.15.149:0/360846940 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb98402ff10 0x7fb9840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:44.714 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.710+0000 7fb999d9b700 1 --2- 172.21.15.149:0/360846940 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb98402ff10 0x7fb9840323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb98c0096a0 tx=0x7fb98c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:44.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.715+0000 7fb98affd700 1 -- 172.21.15.149:0/360846940 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb99001d980 con 0x7fb99c060280 2024-02-12T16:31:44.920 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.918+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/360846940 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb99c108190 con 0x7fb99c060280 2024-02-12T16:31:44.920 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.919+0000 7fb98affd700 1 -- 172.21.15.149:0/360846940 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb990010230 con 0x7fb99c060280 2024-02-12T16:31:44.921 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:44.921 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:44.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.920+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/360846940 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb98402ff10 msgr2=0x7fb9840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:44.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.920+0000 7fb9a0a8f700 1 --2- 172.21.15.149:0/360846940 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb98402ff10 0x7fb9840323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb98c0096a0 tx=0x7fb98c006b40).stop 2024-02-12T16:31:44.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.920+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/360846940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99c060280 msgr2=0x7fb99c106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:44.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.920+0000 7fb9a0a8f700 1 --2- 172.21.15.149:0/360846940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99c060280 0x7fb99c106760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb9900089f0 tx=0x7fb99000b040).stop 2024-02-12T16:31:44.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.920+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/360846940 shutdown_connections 2024-02-12T16:31:44.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.920+0000 7fb9a0a8f700 1 --2- 172.21.15.149:0/360846940 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb98402ff10 0x7fb9840323d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:44.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.920+0000 7fb9a0a8f700 1 --2- 172.21.15.149:0/360846940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb99c060280 0x7fb99c106760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:44.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.920+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/360846940 >> 172.21.15.149:0/360846940 conn(0x7fb99c0f0b70 msgr2=0x7fb99c0f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:44.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.920+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/360846940 shutdown_connections 2024-02-12T16:31:44.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:44.920+0000 7fb9a0a8f700 1 -- 172.21.15.149:0/360846940 wait complete. 2024-02-12T16:31:44.923 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:45.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:45 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:44.919787+0000 mon.smithi104 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.149:0/360846940' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:46.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:44.293721+0000 mgr.smithi104.ohwbhf (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:46.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:46.518 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:48.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.240+0000 7f75d6fee700 1 -- 172.21.15.149:0/2460660549 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d00f5410 msgr2=0x7f75d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:48.248 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.240+0000 7f75d6fee700 1 --2- 172.21.15.149:0/2460660549 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d00f5410 0x7f75d00f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f75c4009fb0 tx=0x7f75c4009b30).stop 2024-02-12T16:31:48.248 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.240+0000 7f75d6fee700 1 -- 172.21.15.149:0/2460660549 shutdown_connections 2024-02-12T16:31:48.248 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.240+0000 7f75d6fee700 1 --2- 172.21.15.149:0/2460660549 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d00f5410 0x7f75d00f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:48.248 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.240+0000 7f75d6fee700 1 -- 172.21.15.149:0/2460660549 >> 172.21.15.149:0/2460660549 conn(0x7f75d00f0b70 msgr2=0x7f75d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:48.248 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.240+0000 7f75d6fee700 1 -- 172.21.15.149:0/2460660549 shutdown_connections 2024-02-12T16:31:48.248 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.240+0000 7f75d6fee700 1 -- 172.21.15.149:0/2460660549 wait complete. 2024-02-12T16:31:48.248 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.240+0000 7f75d6fee700 1 Processor -- start 2024-02-12T16:31:48.248 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.241+0000 7f75d6fee700 1 -- start start 2024-02-12T16:31:48.248 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.241+0000 7f75d6fee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d00f5410 0x7f75d0106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:48.248 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.241+0000 7f75d6fee700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d0106cf0 con 0x7f75d00f5410 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.241+0000 7f75d4d8a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d00f5410 0x7f75d0106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.241+0000 7f75d4d8a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d00f5410 0x7f75d0106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33404/0 (socket says 172.21.15.149:33404) 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.241+0000 7f75d4d8a700 1 -- 172.21.15.149:0/1445119883 learned_addr learned my addr 172.21.15.149:0/1445119883 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.241+0000 7f75d4d8a700 1 -- 172.21.15.149:0/1445119883 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c4005040 con 0x7f75d00f5410 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.242+0000 7f75d4d8a700 1 --2- 172.21.15.149:0/1445119883 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d00f5410 0x7f75d0106800 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f75c40089f0 tx=0x7f75c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.242+0000 7f75cd7fa700 1 -- 172.21.15.149:0/1445119883 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c4009320 con 0x7f75d00f5410 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.242+0000 7f75d6fee700 1 -- 172.21.15.149:0/1445119883 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d0106ef0 con 0x7f75d00f5410 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.242+0000 7f75cd7fa700 1 -- 172.21.15.149:0/1445119883 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75c4013380 con 0x7f75d00f5410 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.242+0000 7f75cd7fa700 1 -- 172.21.15.149:0/1445119883 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c401ed80 con 0x7f75d00f5410 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.242+0000 7f75d6fee700 1 -- 172.21.15.149:0/1445119883 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d0107380 con 0x7f75d00f5410 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.242+0000 7f75cd7fa700 1 -- 172.21.15.149:0/1445119883 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75c4014420 con 0x7f75d00f5410 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.243+0000 7f75cd7fa700 1 --2- 172.21.15.149:0/1445119883 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff60 0x7f75c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.243+0000 7f75cd7fa700 1 -- 172.21.15.149:0/1445119883 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75c4040f70 con 0x7f75d00f5410 2024-02-12T16:31:48.249 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.243+0000 7f75d6fee700 1 -- 172.21.15.149:0/1445119883 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d0059100 con 0x7f75d00f5410 2024-02-12T16:31:48.250 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.243+0000 7f75cffff700 1 --2- 172.21.15.149:0/1445119883 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff60 0x7f75c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:48.250 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.243+0000 7f75cffff700 1 --2- 172.21.15.149:0/1445119883 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff60 0x7f75c0032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f75bc0096a0 tx=0x7f75bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:48.250 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.247+0000 7f75cd7fa700 1 -- 172.21.15.149:0/1445119883 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75c401c910 con 0x7f75d00f5410 2024-02-12T16:31:48.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:46.293970+0000 mgr.smithi104.ohwbhf (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:48.452 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.451+0000 7f75d6fee700 1 -- 172.21.15.149:0/1445119883 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75d00fafa0 con 0x7f75d00f5410 2024-02-12T16:31:48.453 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.452+0000 7f75cd7fa700 1 -- 172.21.15.149:0/1445119883 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f75c401c320 con 0x7f75d00f5410 2024-02-12T16:31:48.453 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:48.453 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:48.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.453+0000 7f75d6fee700 1 -- 172.21.15.149:0/1445119883 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff60 msgr2=0x7f75c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:48.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.453+0000 7f75d6fee700 1 --2- 172.21.15.149:0/1445119883 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff60 0x7f75c0032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f75bc0096a0 tx=0x7f75bc006b40).stop 2024-02-12T16:31:48.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.453+0000 7f75d6fee700 1 -- 172.21.15.149:0/1445119883 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d00f5410 msgr2=0x7f75d0106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:48.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.453+0000 7f75d6fee700 1 --2- 172.21.15.149:0/1445119883 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d00f5410 0x7f75d0106800 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f75c40089f0 tx=0x7f75c400b040).stop 2024-02-12T16:31:48.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.453+0000 7f75d6fee700 1 -- 172.21.15.149:0/1445119883 shutdown_connections 2024-02-12T16:31:48.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.453+0000 7f75d6fee700 1 --2- 172.21.15.149:0/1445119883 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff60 0x7f75c0032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:48.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.453+0000 7f75d6fee700 1 --2- 172.21.15.149:0/1445119883 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d00f5410 0x7f75d0106800 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:48.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.453+0000 7f75d6fee700 1 -- 172.21.15.149:0/1445119883 >> 172.21.15.149:0/1445119883 conn(0x7f75d00f0b70 msgr2=0x7f75d005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:48.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.453+0000 7f75d6fee700 1 -- 172.21.15.149:0/1445119883 shutdown_connections 2024-02-12T16:31:48.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:48.453+0000 7f75d6fee700 1 -- 172.21.15.149:0/1445119883 wait complete. 2024-02-12T16:31:48.456 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:49.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:49 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:48.451936+0000 mon.smithi104 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.149:0/1445119883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:50.104 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:50.105 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:50.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:48.294269+0000 mgr.smithi104.ohwbhf (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:51.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.870+0000 7fa0c42b0700 1 -- 172.21.15.149:0/1603335990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0f5410 msgr2=0x7fa0bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:51.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.870+0000 7fa0c42b0700 1 --2- 172.21.15.149:0/1603335990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0f5410 0x7fa0bc0f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa0ac009fb0 tx=0x7fa0ac009b30).stop 2024-02-12T16:31:51.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.870+0000 7fa0c42b0700 1 -- 172.21.15.149:0/1603335990 shutdown_connections 2024-02-12T16:31:51.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.870+0000 7fa0c42b0700 1 --2- 172.21.15.149:0/1603335990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0f5410 0x7fa0bc0f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:51.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.870+0000 7fa0c42b0700 1 -- 172.21.15.149:0/1603335990 >> 172.21.15.149:0/1603335990 conn(0x7fa0bc0f0b70 msgr2=0x7fa0bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:51.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.870+0000 7fa0c42b0700 1 -- 172.21.15.149:0/1603335990 shutdown_connections 2024-02-12T16:31:51.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.870+0000 7fa0c42b0700 1 -- 172.21.15.149:0/1603335990 wait complete. 2024-02-12T16:31:51.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.871+0000 7fa0c42b0700 1 Processor -- start 2024-02-12T16:31:51.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.871+0000 7fa0c42b0700 1 -- start start 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.871+0000 7fa0c42b0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0f5410 0x7fa0bc109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.871+0000 7fa0c42b0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc060220 con 0x7fa0bc0f5410 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.871+0000 7fa0c204c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0f5410 0x7fa0bc109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.871+0000 7fa0c204c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0f5410 0x7fa0bc109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33430/0 (socket says 172.21.15.149:33430) 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.871+0000 7fa0c204c700 1 -- 172.21.15.149:0/2097582024 learned_addr learned my addr 172.21.15.149:0/2097582024 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.871+0000 7fa0c204c700 1 -- 172.21.15.149:0/2097582024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac005040 con 0x7fa0bc0f5410 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.872+0000 7fa0c204c700 1 --2- 172.21.15.149:0/2097582024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0f5410 0x7fa0bc109500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa0ac005010 tx=0x7fa0ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.872+0000 7fa0b2ffd700 1 -- 172.21.15.149:0/2097582024 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0ac009320 con 0x7fa0bc0f5410 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.872+0000 7fa0c42b0700 1 -- 172.21.15.149:0/2097582024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc060420 con 0x7fa0bc0f5410 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.872+0000 7fa0b2ffd700 1 -- 172.21.15.149:0/2097582024 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0ac013380 con 0x7fa0bc0f5410 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.872+0000 7fa0b2ffd700 1 -- 172.21.15.149:0/2097582024 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0ac014e50 con 0x7fa0bc0f5410 2024-02-12T16:31:51.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.872+0000 7fa0c42b0700 1 -- 172.21.15.149:0/2097582024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc109bb0 con 0x7fa0bc0f5410 2024-02-12T16:31:51.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.873+0000 7fa0b2ffd700 1 -- 172.21.15.149:0/2097582024 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0ac0095f0 con 0x7fa0bc0f5410 2024-02-12T16:31:51.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.873+0000 7fa0c42b0700 1 -- 172.21.15.149:0/2097582024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0bc059100 con 0x7fa0bc0f5410 2024-02-12T16:31:51.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.873+0000 7fa0b2ffd700 1 --2- 172.21.15.149:0/2097582024 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa0a802ff10 0x7fa0a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:51.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.873+0000 7fa0b2ffd700 1 -- 172.21.15.149:0/2097582024 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0ac040e90 con 0x7fa0bc0f5410 2024-02-12T16:31:51.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.873+0000 7fa0c184b700 1 --2- 172.21.15.149:0/2097582024 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa0a802ff10 0x7fa0a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:51.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.874+0000 7fa0c184b700 1 --2- 172.21.15.149:0/2097582024 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa0a802ff10 0x7fa0a80323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa0b80096a0 tx=0x7fa0b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:51.878 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:51.878+0000 7fa0b2ffd700 1 -- 172.21.15.149:0/2097582024 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0ac014420 con 0x7fa0bc0f5410 2024-02-12T16:31:52.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.080+0000 7fa0c42b0700 1 -- 172.21.15.149:0/2097582024 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0bc045ad0 con 0x7fa0bc0f5410 2024-02-12T16:31:52.082 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.081+0000 7fa0b2ffd700 1 -- 172.21.15.149:0/2097582024 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa0ac010230 con 0x7fa0bc0f5410 2024-02-12T16:31:52.082 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:52.083 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:52.084 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.083+0000 7fa0c42b0700 1 -- 172.21.15.149:0/2097582024 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa0a802ff10 msgr2=0x7fa0a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:52.084 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.083+0000 7fa0c42b0700 1 --2- 172.21.15.149:0/2097582024 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa0a802ff10 0x7fa0a80323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa0b80096a0 tx=0x7fa0b8006b40).stop 2024-02-12T16:31:52.084 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.083+0000 7fa0c42b0700 1 -- 172.21.15.149:0/2097582024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0f5410 msgr2=0x7fa0bc109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:52.084 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.083+0000 7fa0c42b0700 1 --2- 172.21.15.149:0/2097582024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0f5410 0x7fa0bc109500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa0ac005010 tx=0x7fa0ac00b040).stop 2024-02-12T16:31:52.084 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.083+0000 7fa0c42b0700 1 -- 172.21.15.149:0/2097582024 shutdown_connections 2024-02-12T16:31:52.084 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.083+0000 7fa0c42b0700 1 --2- 172.21.15.149:0/2097582024 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa0a802ff10 0x7fa0a80323d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:52.084 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.083+0000 7fa0c42b0700 1 --2- 172.21.15.149:0/2097582024 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa0bc0f5410 0x7fa0bc109500 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:52.084 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.083+0000 7fa0c42b0700 1 -- 172.21.15.149:0/2097582024 >> 172.21.15.149:0/2097582024 conn(0x7fa0bc0f0b70 msgr2=0x7fa0bc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:52.085 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.083+0000 7fa0c42b0700 1 -- 172.21.15.149:0/2097582024 shutdown_connections 2024-02-12T16:31:52.085 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:52.083+0000 7fa0c42b0700 1 -- 172.21.15.149:0/2097582024 wait complete. 2024-02-12T16:31:52.085 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:52.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:50.294559+0000 mgr.smithi104.ohwbhf (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:52.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:52.081342+0000 mon.smithi104 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.149:0/2097582024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:53.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:53.759 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:54.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:52.294905+0000 mgr.smithi104.ohwbhf (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:55.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.493+0000 7fc05dc99700 1 -- 172.21.15.149:0/318113117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0580f6510 msgr2=0x7fc0580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:55.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.493+0000 7fc05dc99700 1 --2- 172.21.15.149:0/318113117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0580f6510 0x7fc0580f6930 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc04c009fb0 tx=0x7fc04c009b30).stop 2024-02-12T16:31:55.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.493+0000 7fc05dc99700 1 -- 172.21.15.149:0/318113117 shutdown_connections 2024-02-12T16:31:55.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.493+0000 7fc05dc99700 1 --2- 172.21.15.149:0/318113117 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0580f6510 0x7fc0580f6930 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:55.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.493+0000 7fc05dc99700 1 -- 172.21.15.149:0/318113117 >> 172.21.15.149:0/318113117 conn(0x7fc0580f1c70 msgr2=0x7fc0580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.493+0000 7fc05dc99700 1 -- 172.21.15.149:0/318113117 shutdown_connections 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.493+0000 7fc05dc99700 1 -- 172.21.15.149:0/318113117 wait complete. 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.494+0000 7fc05dc99700 1 Processor -- start 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.494+0000 7fc05dc99700 1 -- start start 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.494+0000 7fc05dc99700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0580f6510 0x7fc0581067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.494+0000 7fc05dc99700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc058106cd0 con 0x7fc0580f6510 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.494+0000 7fc0577fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0580f6510 0x7fc0581067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.494+0000 7fc0577fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0580f6510 0x7fc0581067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55354/0 (socket says 172.21.15.149:55354) 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.494+0000 7fc0577fe700 1 -- 172.21.15.149:0/3145641363 learned_addr learned my addr 172.21.15.149:0/3145641363 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.495+0000 7fc0577fe700 1 -- 172.21.15.149:0/3145641363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c005040 con 0x7fc0580f6510 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.495+0000 7fc0577fe700 1 --2- 172.21.15.149:0/3145641363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0580f6510 0x7fc0581067e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fc04c0089f0 tx=0x7fc04c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.495+0000 7fc03ffff700 1 -- 172.21.15.149:0/3145641363 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc04c009320 con 0x7fc0580f6510 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.495+0000 7fc05dc99700 1 -- 172.21.15.149:0/3145641363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc058106ed0 con 0x7fc0580f6510 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.495+0000 7fc03ffff700 1 -- 172.21.15.149:0/3145641363 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc04c013380 con 0x7fc0580f6510 2024-02-12T16:31:55.499 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.495+0000 7fc03ffff700 1 -- 172.21.15.149:0/3145641363 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc04c01ed80 con 0x7fc0580f6510 2024-02-12T16:31:55.499 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.495+0000 7fc05dc99700 1 -- 172.21.15.149:0/3145641363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc058107360 con 0x7fc0580f6510 2024-02-12T16:31:55.500 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.496+0000 7fc03ffff700 1 -- 172.21.15.149:0/3145641363 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc04c014420 con 0x7fc0580f6510 2024-02-12T16:31:55.500 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.496+0000 7fc03ffff700 1 --2- 172.21.15.149:0/3145641363 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc04002ff60 0x7fc040032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:55.500 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.496+0000 7fc03ffff700 1 -- 172.21.15.149:0/3145641363 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc04c040ed0 con 0x7fc0580f6510 2024-02-12T16:31:55.500 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.496+0000 7fc05dc99700 1 -- 172.21.15.149:0/3145641363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc058045ad0 con 0x7fc0580f6510 2024-02-12T16:31:55.500 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.496+0000 7fc056ffd700 1 --2- 172.21.15.149:0/3145641363 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc04002ff60 0x7fc040032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:55.500 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.497+0000 7fc056ffd700 1 --2- 172.21.15.149:0/3145641363 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc04002ff60 0x7fc040032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc0480096a0 tx=0x7fc048006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:55.503 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.501+0000 7fc03ffff700 1 -- 172.21.15.149:0/3145641363 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc04c01c910 con 0x7fc0580f6510 2024-02-12T16:31:55.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.707+0000 7fc05dc99700 1 -- 172.21.15.149:0/3145641363 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0581080b0 con 0x7fc0580f6510 2024-02-12T16:31:55.710 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.708+0000 7fc03ffff700 1 -- 172.21.15.149:0/3145641363 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc04c01c320 con 0x7fc0580f6510 2024-02-12T16:31:55.710 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:55.710 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:55.711 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.709+0000 7fc05dc99700 1 -- 172.21.15.149:0/3145641363 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc04002ff60 msgr2=0x7fc040032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:55.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.709+0000 7fc05dc99700 1 --2- 172.21.15.149:0/3145641363 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc04002ff60 0x7fc040032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc0480096a0 tx=0x7fc048006b40).stop 2024-02-12T16:31:55.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.709+0000 7fc05dc99700 1 -- 172.21.15.149:0/3145641363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0580f6510 msgr2=0x7fc0581067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:55.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.709+0000 7fc05dc99700 1 --2- 172.21.15.149:0/3145641363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0580f6510 0x7fc0581067e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fc04c0089f0 tx=0x7fc04c00b040).stop 2024-02-12T16:31:55.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.709+0000 7fc05dc99700 1 -- 172.21.15.149:0/3145641363 shutdown_connections 2024-02-12T16:31:55.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.709+0000 7fc05dc99700 1 --2- 172.21.15.149:0/3145641363 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc04002ff60 0x7fc040032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:55.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.709+0000 7fc05dc99700 1 --2- 172.21.15.149:0/3145641363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0580f6510 0x7fc0581067e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:55.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.709+0000 7fc05dc99700 1 -- 172.21.15.149:0/3145641363 >> 172.21.15.149:0/3145641363 conn(0x7fc0580f1c70 msgr2=0x7fc0580f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:55.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.709+0000 7fc05dc99700 1 -- 172.21.15.149:0/3145641363 shutdown_connections 2024-02-12T16:31:55.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:55.709+0000 7fc05dc99700 1 -- 172.21.15.149:0/3145641363 wait complete. 2024-02-12T16:31:55.712 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:31:56.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:54.295219+0000 mgr.smithi104.ohwbhf (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:56.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:55.708909+0000 mon.smithi104 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.149:0/3145641363' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:31:57.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:31:57.327 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:31:58.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:31:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:56.295495+0000 mgr.smithi104.ohwbhf (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:31:59.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.155+0000 7fa531a6d700 1 -- 172.21.15.149:0/2897505649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa52c060280 msgr2=0x7fa52c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:59.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.155+0000 7fa531a6d700 1 --2- 172.21.15.149:0/2897505649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa52c060280 0x7fa52c0f90f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa514009fb0 tx=0x7fa514009b30).stop 2024-02-12T16:31:59.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.155+0000 7fa531a6d700 1 -- 172.21.15.149:0/2897505649 shutdown_connections 2024-02-12T16:31:59.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.155+0000 7fa531a6d700 1 --2- 172.21.15.149:0/2897505649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa52c060280 0x7fa52c0f90f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:59.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.155+0000 7fa531a6d700 1 -- 172.21.15.149:0/2897505649 >> 172.21.15.149:0/2897505649 conn(0x7fa52c0f0b70 msgr2=0x7fa52c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:59.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.155+0000 7fa531a6d700 1 -- 172.21.15.149:0/2897505649 shutdown_connections 2024-02-12T16:31:59.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.155+0000 7fa531a6d700 1 -- 172.21.15.149:0/2897505649 wait complete. 2024-02-12T16:31:59.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.155+0000 7fa531a6d700 1 Processor -- start 2024-02-12T16:31:59.159 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.156+0000 7fa531a6d700 1 -- start start 2024-02-12T16:31:59.159 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.156+0000 7fa531a6d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa52c060280 0x7fa52c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:59.159 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.156+0000 7fa531a6d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c0f4ff0 con 0x7fa52c060280 2024-02-12T16:31:59.159 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.156+0000 7fa52b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa52c060280 0x7fa52c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:59.159 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.156+0000 7fa52b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa52c060280 0x7fa52c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55364/0 (socket says 172.21.15.149:55364) 2024-02-12T16:31:59.159 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.156+0000 7fa52b7fe700 1 -- 172.21.15.149:0/2015091254 learned_addr learned my addr 172.21.15.149:0/2015091254 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:31:59.160 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.156+0000 7fa52b7fe700 1 -- 172.21.15.149:0/2015091254 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa514005040 con 0x7fa52c060280 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.156+0000 7fa52b7fe700 1 --2- 172.21.15.149:0/2015091254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa52c060280 0x7fa52c0f6b70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fa5140089f0 tx=0x7fa51400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.157+0000 7fa523fff700 1 -- 172.21.15.149:0/2015091254 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa514009320 con 0x7fa52c060280 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.157+0000 7fa523fff700 1 -- 172.21.15.149:0/2015091254 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa514013380 con 0x7fa52c060280 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.157+0000 7fa531a6d700 1 -- 172.21.15.149:0/2015091254 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52c0f5190 con 0x7fa52c060280 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.157+0000 7fa523fff700 1 -- 172.21.15.149:0/2015091254 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa514014e20 con 0x7fa52c060280 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.157+0000 7fa531a6d700 1 -- 172.21.15.149:0/2015091254 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52c0f5620 con 0x7fa52c060280 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.157+0000 7fa523fff700 1 -- 172.21.15.149:0/2015091254 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5140095f0 con 0x7fa52c060280 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.157+0000 7fa531a6d700 1 -- 172.21.15.149:0/2015091254 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52c059100 con 0x7fa52c060280 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.158+0000 7fa523fff700 1 --2- 172.21.15.149:0/2015091254 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa51802ff60 0x7fa518032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.158+0000 7fa523fff700 1 -- 172.21.15.149:0/2015091254 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa514040f00 con 0x7fa52c060280 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.158+0000 7fa52affd700 1 --2- 172.21.15.149:0/2015091254 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa51802ff60 0x7fa518032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:31:59.161 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.158+0000 7fa52affd700 1 --2- 172.21.15.149:0/2015091254 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa51802ff60 0x7fa518032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa51c0096a0 tx=0x7fa51c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:31:59.164 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.162+0000 7fa523fff700 1 -- 172.21.15.149:0/2015091254 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa514014420 con 0x7fa52c060280 2024-02-12T16:31:59.368 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.366+0000 7fa531a6d700 1 -- 172.21.15.149:0/2015091254 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa52c0fc090 con 0x7fa52c060280 2024-02-12T16:31:59.368 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.366+0000 7fa523fff700 1 -- 172.21.15.149:0/2015091254 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa514010230 con 0x7fa52c060280 2024-02-12T16:31:59.368 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:31:59.369 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:31:59.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.368+0000 7fa531a6d700 1 -- 172.21.15.149:0/2015091254 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa51802ff60 msgr2=0x7fa518032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:59.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.368+0000 7fa531a6d700 1 --2- 172.21.15.149:0/2015091254 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa51802ff60 0x7fa518032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa51c0096a0 tx=0x7fa51c006b40).stop 2024-02-12T16:31:59.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.368+0000 7fa531a6d700 1 -- 172.21.15.149:0/2015091254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa52c060280 msgr2=0x7fa52c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:31:59.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.368+0000 7fa531a6d700 1 --2- 172.21.15.149:0/2015091254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa52c060280 0x7fa52c0f6b70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fa5140089f0 tx=0x7fa51400b040).stop 2024-02-12T16:31:59.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.368+0000 7fa531a6d700 1 -- 172.21.15.149:0/2015091254 shutdown_connections 2024-02-12T16:31:59.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.368+0000 7fa531a6d700 1 --2- 172.21.15.149:0/2015091254 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa51802ff60 0x7fa518032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:59.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.368+0000 7fa531a6d700 1 --2- 172.21.15.149:0/2015091254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa52c060280 0x7fa52c0f6b70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:31:59.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.368+0000 7fa531a6d700 1 -- 172.21.15.149:0/2015091254 >> 172.21.15.149:0/2015091254 conn(0x7fa52c0f0b70 msgr2=0x7fa52c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:31:59.371 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.368+0000 7fa531a6d700 1 -- 172.21.15.149:0/2015091254 shutdown_connections 2024-02-12T16:31:59.371 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:31:59.368+0000 7fa531a6d700 1 -- 172.21.15.149:0/2015091254 wait complete. 2024-02-12T16:31:59.371 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:00.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:31:58.295843+0000 mgr.smithi104.ohwbhf (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:00.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:31:59.367538+0000 mon.smithi104 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.149:0/2015091254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:01.035 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:01.035 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:02.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:00.296087+0000 mgr.smithi104.ohwbhf (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:02.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.870+0000 7fe177eec700 1 -- 172.21.15.149:0/251828503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1700f6510 msgr2=0x7fe1700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:02.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.870+0000 7fe177eec700 1 --2- 172.21.15.149:0/251828503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1700f6510 0x7fe1700f6930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fe160009fb0 tx=0x7fe160009b30).stop 2024-02-12T16:32:02.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.870+0000 7fe177eec700 1 -- 172.21.15.149:0/251828503 shutdown_connections 2024-02-12T16:32:02.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.870+0000 7fe177eec700 1 --2- 172.21.15.149:0/251828503 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1700f6510 0x7fe1700f6930 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:02.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.870+0000 7fe177eec700 1 -- 172.21.15.149:0/251828503 >> 172.21.15.149:0/251828503 conn(0x7fe1700f1c70 msgr2=0x7fe1700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:02.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.870+0000 7fe177eec700 1 -- 172.21.15.149:0/251828503 shutdown_connections 2024-02-12T16:32:02.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.870+0000 7fe177eec700 1 -- 172.21.15.149:0/251828503 wait complete. 2024-02-12T16:32:02.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.871+0000 7fe177eec700 1 Processor -- start 2024-02-12T16:32:02.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.871+0000 7fe177eec700 1 -- start start 2024-02-12T16:32:02.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.871+0000 7fe177eec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1700f6510 0x7fe17006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:02.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.871+0000 7fe177eec700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe170069e20 con 0x7fe1700f6510 2024-02-12T16:32:02.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.871+0000 7fe175c88700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1700f6510 0x7fe17006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:02.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.871+0000 7fe175c88700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1700f6510 0x7fe17006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55380/0 (socket says 172.21.15.149:55380) 2024-02-12T16:32:02.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.871+0000 7fe175c88700 1 -- 172.21.15.149:0/2892145062 learned_addr learned my addr 172.21.15.149:0/2892145062 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:02.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.871+0000 7fe175c88700 1 -- 172.21.15.149:0/2892145062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe160005040 con 0x7fe1700f6510 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.872+0000 7fe175c88700 1 --2- 172.21.15.149:0/2892145062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1700f6510 0x7fe17006b9a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fe160005010 tx=0x7fe16000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.872+0000 7fe1667fc700 1 -- 172.21.15.149:0/2892145062 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe160009320 con 0x7fe1700f6510 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.872+0000 7fe177eec700 1 -- 172.21.15.149:0/2892145062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe170069fc0 con 0x7fe1700f6510 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.872+0000 7fe1667fc700 1 -- 172.21.15.149:0/2892145062 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe160013380 con 0x7fe1700f6510 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.872+0000 7fe1667fc700 1 -- 172.21.15.149:0/2892145062 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe160014e50 con 0x7fe1700f6510 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.872+0000 7fe177eec700 1 -- 172.21.15.149:0/2892145062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17006a450 con 0x7fe1700f6510 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.873+0000 7fe1667fc700 1 -- 172.21.15.149:0/2892145062 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1600095f0 con 0x7fe1700f6510 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.873+0000 7fe177eec700 1 -- 172.21.15.149:0/2892145062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe170059100 con 0x7fe1700f6510 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.873+0000 7fe1667fc700 1 --2- 172.21.15.149:0/2892145062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 0x7fe15c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.873+0000 7fe1667fc700 1 -- 172.21.15.149:0/2892145062 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe160040f10 con 0x7fe1700f6510 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.873+0000 7fe175487700 1 --2- 172.21.15.149:0/2892145062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 0x7fe15c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:02.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.873+0000 7fe175487700 1 --2- 172.21.15.149:0/2892145062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 0x7fe15c0323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fe16c0096a0 tx=0x7fe16c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:02.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:02.878+0000 7fe1667fc700 1 -- 172.21.15.149:0/2892145062 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe160014420 con 0x7fe1700f6510 2024-02-12T16:32:03.083 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.081+0000 7fe177eec700 1 -- 172.21.15.149:0/2892145062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1700fc0a0 con 0x7fe1700f6510 2024-02-12T16:32:03.083 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.081+0000 7fe1667fc700 1 -- 172.21.15.149:0/2892145062 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe160010230 con 0x7fe1700f6510 2024-02-12T16:32:03.084 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:03.084 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:03.085 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.083+0000 7fe177eec700 1 -- 172.21.15.149:0/2892145062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 msgr2=0x7fe15c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:03.086 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.083+0000 7fe177eec700 1 --2- 172.21.15.149:0/2892145062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 0x7fe15c0323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fe16c0096a0 tx=0x7fe16c006b40).stop 2024-02-12T16:32:03.086 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.083+0000 7fe177eec700 1 -- 172.21.15.149:0/2892145062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1700f6510 msgr2=0x7fe17006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:03.086 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.083+0000 7fe177eec700 1 --2- 172.21.15.149:0/2892145062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1700f6510 0x7fe17006b9a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fe160005010 tx=0x7fe16000b040).stop 2024-02-12T16:32:03.086 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.083+0000 7fe177eec700 1 -- 172.21.15.149:0/2892145062 shutdown_connections 2024-02-12T16:32:03.086 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.083+0000 7fe177eec700 1 --2- 172.21.15.149:0/2892145062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 0x7fe15c0323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:03.086 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.083+0000 7fe177eec700 1 --2- 172.21.15.149:0/2892145062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1700f6510 0x7fe17006b9a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:03.086 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.083+0000 7fe177eec700 1 -- 172.21.15.149:0/2892145062 >> 172.21.15.149:0/2892145062 conn(0x7fe1700f1c70 msgr2=0x7fe1700f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:03.086 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.083+0000 7fe177eec700 1 -- 172.21.15.149:0/2892145062 shutdown_connections 2024-02-12T16:32:03.086 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:03.083+0000 7fe177eec700 1 -- 172.21.15.149:0/2892145062 wait complete. 2024-02-12T16:32:03.087 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:03.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:03 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:03.082723+0000 mon.smithi104 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.149:0/2892145062' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:04.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:02.296425+0000 mgr.smithi104.ohwbhf (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:04.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:04.720 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:06.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:04.296691+0000 mgr.smithi104.ohwbhf (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:06.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.504+0000 7f74b8a17700 1 -- 172.21.15.149:0/3146250164 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74b0013990 con 0x7f74b40f6510 2024-02-12T16:32:06.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.505+0000 7f74bbc7d700 1 -- 172.21.15.149:0/3146250164 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b40f6510 msgr2=0x7f74b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:06.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.505+0000 7f74bbc7d700 1 --2- 172.21.15.149:0/3146250164 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b40f6510 0x7f74b40f6930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f74b0009fb0 tx=0x7f74b0009b30).stop 2024-02-12T16:32:06.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.505+0000 7f74bbc7d700 1 -- 172.21.15.149:0/3146250164 shutdown_connections 2024-02-12T16:32:06.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.505+0000 7f74bbc7d700 1 --2- 172.21.15.149:0/3146250164 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b40f6510 0x7f74b40f6930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:06.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.505+0000 7f74bbc7d700 1 -- 172.21.15.149:0/3146250164 >> 172.21.15.149:0/3146250164 conn(0x7f74b40f1c70 msgr2=0x7f74b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:06.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.505+0000 7f74bbc7d700 1 -- 172.21.15.149:0/3146250164 shutdown_connections 2024-02-12T16:32:06.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.505+0000 7f74bbc7d700 1 -- 172.21.15.149:0/3146250164 wait complete. 2024-02-12T16:32:06.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.506+0000 7f74bbc7d700 1 Processor -- start 2024-02-12T16:32:06.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.506+0000 7f74bbc7d700 1 -- start start 2024-02-12T16:32:06.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.506+0000 7f74bbc7d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b40f6510 0x7f74b41089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:06.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.506+0000 7f74bbc7d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b4108ee0 con 0x7f74b40f6510 2024-02-12T16:32:06.510 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.506+0000 7f74b9a19700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b40f6510 0x7f74b41089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:06.510 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.506+0000 7f74b9a19700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b40f6510 0x7f74b41089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:41178/0 (socket says 172.21.15.149:41178) 2024-02-12T16:32:06.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.506+0000 7f74b9a19700 1 -- 172.21.15.149:0/756974432 learned_addr learned my addr 172.21.15.149:0/756974432 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:06.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.506+0000 7f74b9a19700 1 -- 172.21.15.149:0/756974432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b0005040 con 0x7f74b40f6510 2024-02-12T16:32:06.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.507+0000 7f74b9a19700 1 --2- 172.21.15.149:0/756974432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b40f6510 0x7f74b41089f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f74b0005010 tx=0x7f74b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:06.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.507+0000 7f74aa7fc700 1 -- 172.21.15.149:0/756974432 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74b0009320 con 0x7f74b40f6510 2024-02-12T16:32:06.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.507+0000 7f74bbc7d700 1 -- 172.21.15.149:0/756974432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b41090e0 con 0x7f74b40f6510 2024-02-12T16:32:06.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.507+0000 7f74aa7fc700 1 -- 172.21.15.149:0/756974432 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74b0013380 con 0x7f74b40f6510 2024-02-12T16:32:06.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.507+0000 7f74aa7fc700 1 -- 172.21.15.149:0/756974432 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74b0014e50 con 0x7f74b40f6510 2024-02-12T16:32:06.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.507+0000 7f74bbc7d700 1 -- 172.21.15.149:0/756974432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b4109570 con 0x7f74b40f6510 2024-02-12T16:32:06.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.508+0000 7f74aa7fc700 1 -- 172.21.15.149:0/756974432 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74b0014420 con 0x7f74b40f6510 2024-02-12T16:32:06.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.508+0000 7f74bbc7d700 1 -- 172.21.15.149:0/756974432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b4045ad0 con 0x7f74b40f6510 2024-02-12T16:32:06.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.508+0000 7f74aa7fc700 1 --2- 172.21.15.149:0/756974432 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f74a002ff10 0x7f74a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:06.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.508+0000 7f74aa7fc700 1 -- 172.21.15.149:0/756974432 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74b0040e60 con 0x7f74b40f6510 2024-02-12T16:32:06.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.508+0000 7f74b9218700 1 --2- 172.21.15.149:0/756974432 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f74a002ff10 0x7f74a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:06.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.509+0000 7f74b9218700 1 --2- 172.21.15.149:0/756974432 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f74a002ff10 0x7f74a00323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f74a40096a0 tx=0x7f74a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:06.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.513+0000 7f74aa7fc700 1 -- 172.21.15.149:0/756974432 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74b0012330 con 0x7f74b40f6510 2024-02-12T16:32:06.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.719+0000 7f74bbc7d700 1 -- 172.21.15.149:0/756974432 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74b406b150 con 0x7f74b40f6510 2024-02-12T16:32:06.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.719+0000 7f74aa7fc700 1 -- 172.21.15.149:0/756974432 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f74b001db70 con 0x7f74b40f6510 2024-02-12T16:32:06.722 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:06.722 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:06.723 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.721+0000 7f74bbc7d700 1 -- 172.21.15.149:0/756974432 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f74a002ff10 msgr2=0x7f74a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:06.723 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.721+0000 7f74bbc7d700 1 --2- 172.21.15.149:0/756974432 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f74a002ff10 0x7f74a00323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f74a40096a0 tx=0x7f74a4006b40).stop 2024-02-12T16:32:06.723 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.721+0000 7f74bbc7d700 1 -- 172.21.15.149:0/756974432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b40f6510 msgr2=0x7f74b41089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:06.723 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.721+0000 7f74bbc7d700 1 --2- 172.21.15.149:0/756974432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b40f6510 0x7f74b41089f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f74b0005010 tx=0x7f74b000b040).stop 2024-02-12T16:32:06.723 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.721+0000 7f74bbc7d700 1 -- 172.21.15.149:0/756974432 shutdown_connections 2024-02-12T16:32:06.724 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.721+0000 7f74bbc7d700 1 --2- 172.21.15.149:0/756974432 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f74a002ff10 0x7f74a00323d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:06.724 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.721+0000 7f74bbc7d700 1 --2- 172.21.15.149:0/756974432 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74b40f6510 0x7f74b41089f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:06.724 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.721+0000 7f74bbc7d700 1 -- 172.21.15.149:0/756974432 >> 172.21.15.149:0/756974432 conn(0x7f74b40f1c70 msgr2=0x7f74b40f2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:06.724 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.721+0000 7f74bbc7d700 1 -- 172.21.15.149:0/756974432 shutdown_connections 2024-02-12T16:32:06.724 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:06.721+0000 7f74bbc7d700 1 -- 172.21.15.149:0/756974432 wait complete. 2024-02-12T16:32:06.724 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:07.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:07 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:06.720712+0000 mon.smithi104 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.149:0/756974432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:08.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:08.338 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:08.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:06.296927+0000 mgr.smithi104.ohwbhf (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:10.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.115+0000 7f5b4e382700 1 -- 172.21.15.149:0/17522996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b480f5410 msgr2=0x7f5b480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:10.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.115+0000 7f5b4e382700 1 --2- 172.21.15.149:0/17522996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b480f5410 0x7f5b480f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f5b38009fb0 tx=0x7f5b38009b30).stop 2024-02-12T16:32:10.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.115+0000 7f5b4e382700 1 -- 172.21.15.149:0/17522996 shutdown_connections 2024-02-12T16:32:10.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.115+0000 7f5b4e382700 1 --2- 172.21.15.149:0/17522996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b480f5410 0x7f5b480f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:10.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.115+0000 7f5b4e382700 1 -- 172.21.15.149:0/17522996 >> 172.21.15.149:0/17522996 conn(0x7f5b480f0b70 msgr2=0x7f5b480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.115+0000 7f5b4e382700 1 -- 172.21.15.149:0/17522996 shutdown_connections 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.115+0000 7f5b4e382700 1 -- 172.21.15.149:0/17522996 wait complete. 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.116+0000 7f5b4e382700 1 Processor -- start 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.116+0000 7f5b4e382700 1 -- start start 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.116+0000 7f5b4e382700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b480f5410 0x7f5b481067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.116+0000 7f5b4e382700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b48106cc0 con 0x7f5b480f5410 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.116+0000 7f5b47fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b480f5410 0x7f5b481067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.116+0000 7f5b47fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b480f5410 0x7f5b481067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:41186/0 (socket says 172.21.15.149:41186) 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.116+0000 7f5b47fff700 1 -- 172.21.15.149:0/2754785496 learned_addr learned my addr 172.21.15.149:0/2754785496 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.117+0000 7f5b47fff700 1 -- 172.21.15.149:0/2754785496 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b38005040 con 0x7f5b480f5410 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.117+0000 7f5b47fff700 1 --2- 172.21.15.149:0/2754785496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b480f5410 0x7f5b481067d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f5b38005010 tx=0x7f5b3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.117+0000 7f5b44ff9700 1 -- 172.21.15.149:0/2754785496 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b38009320 con 0x7f5b480f5410 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.117+0000 7f5b4e382700 1 -- 172.21.15.149:0/2754785496 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b48106ec0 con 0x7f5b480f5410 2024-02-12T16:32:10.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.117+0000 7f5b44ff9700 1 -- 172.21.15.149:0/2754785496 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b38013380 con 0x7f5b480f5410 2024-02-12T16:32:10.121 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.117+0000 7f5b44ff9700 1 -- 172.21.15.149:0/2754785496 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b38014e50 con 0x7f5b480f5410 2024-02-12T16:32:10.121 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.117+0000 7f5b4e382700 1 -- 172.21.15.149:0/2754785496 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b48107350 con 0x7f5b480f5410 2024-02-12T16:32:10.121 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.118+0000 7f5b44ff9700 1 -- 172.21.15.149:0/2754785496 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b380263a0 con 0x7f5b480f5410 2024-02-12T16:32:10.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.118+0000 7f5b4e382700 1 -- 172.21.15.149:0/2754785496 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b48045ad0 con 0x7f5b480f5410 2024-02-12T16:32:10.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.118+0000 7f5b44ff9700 1 --2- 172.21.15.149:0/2754785496 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b3002ff10 0x7f5b300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:10.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.118+0000 7f5b44ff9700 1 -- 172.21.15.149:0/2754785496 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b38041190 con 0x7f5b480f5410 2024-02-12T16:32:10.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.118+0000 7f5b477fe700 1 --2- 172.21.15.149:0/2754785496 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b3002ff10 0x7f5b300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:10.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.118+0000 7f5b477fe700 1 --2- 172.21.15.149:0/2754785496 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b3002ff10 0x7f5b300323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f5b3c0096a0 tx=0x7f5b3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:10.124 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.122+0000 7f5b44ff9700 1 -- 172.21.15.149:0/2754785496 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b380126f0 con 0x7f5b480f5410 2024-02-12T16:32:10.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.327+0000 7f5b4e382700 1 -- 172.21.15.149:0/2754785496 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b481081a0 con 0x7f5b480f5410 2024-02-12T16:32:10.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.328+0000 7f5b44ff9700 1 -- 172.21.15.149:0/2754785496 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5b3800f340 con 0x7f5b480f5410 2024-02-12T16:32:10.330 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:10.330 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:10.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.329+0000 7f5b4e382700 1 -- 172.21.15.149:0/2754785496 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b3002ff10 msgr2=0x7f5b300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:10.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.329+0000 7f5b4e382700 1 --2- 172.21.15.149:0/2754785496 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b3002ff10 0x7f5b300323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f5b3c0096a0 tx=0x7f5b3c006b40).stop 2024-02-12T16:32:10.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.329+0000 7f5b4e382700 1 -- 172.21.15.149:0/2754785496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b480f5410 msgr2=0x7f5b481067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:10.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.329+0000 7f5b4e382700 1 --2- 172.21.15.149:0/2754785496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b480f5410 0x7f5b481067d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f5b38005010 tx=0x7f5b3800b040).stop 2024-02-12T16:32:10.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.330+0000 7f5b4e382700 1 -- 172.21.15.149:0/2754785496 shutdown_connections 2024-02-12T16:32:10.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.330+0000 7f5b4e382700 1 --2- 172.21.15.149:0/2754785496 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5b3002ff10 0x7f5b300323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:10.333 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.330+0000 7f5b4e382700 1 --2- 172.21.15.149:0/2754785496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b480f5410 0x7f5b481067d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:10.333 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.330+0000 7f5b4e382700 1 -- 172.21.15.149:0/2754785496 >> 172.21.15.149:0/2754785496 conn(0x7f5b480f0b70 msgr2=0x7f5b4805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:10.333 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.330+0000 7f5b4e382700 1 -- 172.21.15.149:0/2754785496 shutdown_connections 2024-02-12T16:32:10.333 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:10.330+0000 7f5b4e382700 1 -- 172.21.15.149:0/2754785496 wait complete. 2024-02-12T16:32:10.333 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:10.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:08.297208+0000 mgr.smithi104.ohwbhf (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:11.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:10.329260+0000 mon.smithi104 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.149:0/2754785496' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:11.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:11.926 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:12.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:10.297456+0000 mgr.smithi104.ohwbhf (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:13.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.682+0000 7f503b533700 1 -- 172.21.15.149:0/2407271812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50340f62a0 msgr2=0x7f50340f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:13.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.682+0000 7f503b533700 1 --2- 172.21.15.149:0/2407271812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50340f62a0 0x7f50340f66c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f5024009fb0 tx=0x7f5024009b30).stop 2024-02-12T16:32:13.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.682+0000 7f503b533700 1 -- 172.21.15.149:0/2407271812 shutdown_connections 2024-02-12T16:32:13.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.682+0000 7f503b533700 1 --2- 172.21.15.149:0/2407271812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50340f62a0 0x7f50340f66c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:13.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.682+0000 7f503b533700 1 -- 172.21.15.149:0/2407271812 >> 172.21.15.149:0/2407271812 conn(0x7f50340f1aa0 msgr2=0x7f50340f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:13.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.682+0000 7f503b533700 1 -- 172.21.15.149:0/2407271812 shutdown_connections 2024-02-12T16:32:13.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.682+0000 7f503b533700 1 -- 172.21.15.149:0/2407271812 wait complete. 2024-02-12T16:32:13.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.683+0000 7f503b533700 1 Processor -- start 2024-02-12T16:32:13.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.683+0000 7f503b533700 1 -- start start 2024-02-12T16:32:13.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.683+0000 7f503b533700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50340f62a0 0x7f503406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:13.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.683+0000 7f503b533700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5034069e20 con 0x7f50340f62a0 2024-02-12T16:32:13.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.683+0000 7f503a531700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50340f62a0 0x7f503406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:13.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.683+0000 7f503a531700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50340f62a0 0x7f503406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:41204/0 (socket says 172.21.15.149:41204) 2024-02-12T16:32:13.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.683+0000 7f503a531700 1 -- 172.21.15.149:0/3829396328 learned_addr learned my addr 172.21.15.149:0/3829396328 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:13.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.684+0000 7f503a531700 1 -- 172.21.15.149:0/3829396328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5024005040 con 0x7f50340f62a0 2024-02-12T16:32:13.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.684+0000 7f503a531700 1 --2- 172.21.15.149:0/3829396328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50340f62a0 0x7f503406b9a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f50240089f0 tx=0x7f502400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:13.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.684+0000 7f502affd700 1 -- 172.21.15.149:0/3829396328 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5024009320 con 0x7f50340f62a0 2024-02-12T16:32:13.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.684+0000 7f503b533700 1 -- 172.21.15.149:0/3829396328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5034069fc0 con 0x7f50340f62a0 2024-02-12T16:32:13.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.684+0000 7f503b533700 1 -- 172.21.15.149:0/3829396328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503406a450 con 0x7f50340f62a0 2024-02-12T16:32:13.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.684+0000 7f502affd700 1 -- 172.21.15.149:0/3829396328 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5024013380 con 0x7f50340f62a0 2024-02-12T16:32:13.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.684+0000 7f502affd700 1 -- 172.21.15.149:0/3829396328 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5024014e40 con 0x7f50340f62a0 2024-02-12T16:32:13.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.685+0000 7f502affd700 1 -- 172.21.15.149:0/3829396328 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5024014420 con 0x7f50340f62a0 2024-02-12T16:32:13.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.685+0000 7f503b533700 1 -- 172.21.15.149:0/3829396328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5034059100 con 0x7f50340f62a0 2024-02-12T16:32:13.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.685+0000 7f502affd700 1 --2- 172.21.15.149:0/3829396328 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f502002ff10 0x7f50200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:13.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.685+0000 7f502affd700 1 -- 172.21.15.149:0/3829396328 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5024041220 con 0x7f50340f62a0 2024-02-12T16:32:13.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.685+0000 7f5039d30700 1 --2- 172.21.15.149:0/3829396328 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f502002ff10 0x7f50200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:13.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.685+0000 7f5039d30700 1 --2- 172.21.15.149:0/3829396328 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f502002ff10 0x7f50200323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f50300096a0 tx=0x7f5030006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:13.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.690+0000 7f502affd700 1 -- 172.21.15.149:0/3829396328 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50240127a0 con 0x7f50340f62a0 2024-02-12T16:32:13.895 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.893+0000 7f503b533700 1 -- 172.21.15.149:0/3829396328 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50340fbe30 con 0x7f50340f62a0 2024-02-12T16:32:13.896 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.894+0000 7f502affd700 1 -- 172.21.15.149:0/3829396328 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f50240127a0 con 0x7f50340f62a0 2024-02-12T16:32:13.897 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:13.897 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:13.898 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.896+0000 7f503b533700 1 -- 172.21.15.149:0/3829396328 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f502002ff10 msgr2=0x7f50200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:13.898 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.896+0000 7f503b533700 1 --2- 172.21.15.149:0/3829396328 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f502002ff10 0x7f50200323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f50300096a0 tx=0x7f5030006b40).stop 2024-02-12T16:32:13.898 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.896+0000 7f503b533700 1 -- 172.21.15.149:0/3829396328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50340f62a0 msgr2=0x7f503406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:13.898 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.896+0000 7f503b533700 1 --2- 172.21.15.149:0/3829396328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50340f62a0 0x7f503406b9a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f50240089f0 tx=0x7f502400b040).stop 2024-02-12T16:32:13.898 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.896+0000 7f503b533700 1 -- 172.21.15.149:0/3829396328 shutdown_connections 2024-02-12T16:32:13.898 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.896+0000 7f503b533700 1 --2- 172.21.15.149:0/3829396328 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f502002ff10 0x7f50200323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:13.898 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.896+0000 7f503b533700 1 --2- 172.21.15.149:0/3829396328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50340f62a0 0x7f503406b9a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:13.898 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.896+0000 7f503b533700 1 -- 172.21.15.149:0/3829396328 >> 172.21.15.149:0/3829396328 conn(0x7f50340f1aa0 msgr2=0x7f50340f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:13.899 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.896+0000 7f503b533700 1 -- 172.21.15.149:0/3829396328 shutdown_connections 2024-02-12T16:32:13.899 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:13.896+0000 7f503b533700 1 -- 172.21.15.149:0/3829396328 wait complete. 2024-02-12T16:32:13.899 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:14.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:12.297728+0000 mgr.smithi104.ohwbhf (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:14.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:13.895230+0000 mon.smithi104 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.149:0/3829396328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:15.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:15.535 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:16.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:14.298065+0000 mgr.smithi104.ohwbhf (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:17.343 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.340+0000 7fcf73d63700 1 -- 172.21.15.149:0/3757801685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf6c0f5410 msgr2=0x7fcf6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:17.343 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.340+0000 7fcf73d63700 1 --2- 172.21.15.149:0/3757801685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf6c0f5410 0x7fcf6c0f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fcf5c009fb0 tx=0x7fcf5c009b30).stop 2024-02-12T16:32:17.343 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.340+0000 7fcf73d63700 1 -- 172.21.15.149:0/3757801685 shutdown_connections 2024-02-12T16:32:17.343 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.340+0000 7fcf73d63700 1 --2- 172.21.15.149:0/3757801685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf6c0f5410 0x7fcf6c0f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:17.343 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.340+0000 7fcf73d63700 1 -- 172.21.15.149:0/3757801685 >> 172.21.15.149:0/3757801685 conn(0x7fcf6c0f0b70 msgr2=0x7fcf6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:17.343 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.340+0000 7fcf73d63700 1 -- 172.21.15.149:0/3757801685 shutdown_connections 2024-02-12T16:32:17.343 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.340+0000 7fcf73d63700 1 -- 172.21.15.149:0/3757801685 wait complete. 2024-02-12T16:32:17.343 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.341+0000 7fcf73d63700 1 Processor -- start 2024-02-12T16:32:17.343 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.341+0000 7fcf73d63700 1 -- start start 2024-02-12T16:32:17.344 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.341+0000 7fcf73d63700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf6c0f5410 0x7fcf6c10b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:17.344 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.341+0000 7fcf73d63700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c1072d0 con 0x7fcf6c0f5410 2024-02-12T16:32:17.345 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.341+0000 7fcf71aff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf6c0f5410 0x7fcf6c10b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.341+0000 7fcf71aff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf6c0f5410 0x7fcf6c10b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:58448/0 (socket says 172.21.15.149:58448) 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.341+0000 7fcf71aff700 1 -- 172.21.15.149:0/2055389644 learned_addr learned my addr 172.21.15.149:0/2055389644 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.342+0000 7fcf71aff700 1 -- 172.21.15.149:0/2055389644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c005040 con 0x7fcf6c0f5410 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.342+0000 7fcf71aff700 1 --2- 172.21.15.149:0/2055389644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf6c0f5410 0x7fcf6c10b8e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fcf5c005010 tx=0x7fcf5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.342+0000 7fcf627fc700 1 -- 172.21.15.149:0/2055389644 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf5c009320 con 0x7fcf6c0f5410 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.342+0000 7fcf73d63700 1 -- 172.21.15.149:0/2055389644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6c060240 con 0x7fcf6c0f5410 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.342+0000 7fcf627fc700 1 -- 172.21.15.149:0/2055389644 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf5c013380 con 0x7fcf6c0f5410 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.342+0000 7fcf627fc700 1 -- 172.21.15.149:0/2055389644 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf5c014e50 con 0x7fcf6c0f5410 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.342+0000 7fcf73d63700 1 -- 172.21.15.149:0/2055389644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c10bdd0 con 0x7fcf6c0f5410 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.343+0000 7fcf627fc700 1 -- 172.21.15.149:0/2055389644 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf5c014420 con 0x7fcf6c0f5410 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.343+0000 7fcf73d63700 1 -- 172.21.15.149:0/2055389644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6c045ad0 con 0x7fcf6c0f5410 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.343+0000 7fcf627fc700 1 --2- 172.21.15.149:0/2055389644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcf5802ff10 0x7fcf580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:17.346 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.343+0000 7fcf627fc700 1 -- 172.21.15.149:0/2055389644 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf5c040f90 con 0x7fcf6c0f5410 2024-02-12T16:32:17.347 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.343+0000 7fcf712fe700 1 --2- 172.21.15.149:0/2055389644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcf5802ff10 0x7fcf580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:17.347 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.343+0000 7fcf712fe700 1 --2- 172.21.15.149:0/2055389644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcf5802ff10 0x7fcf580323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcf680096a0 tx=0x7fcf68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:17.349 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.347+0000 7fcf627fc700 1 -- 172.21.15.149:0/2055389644 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf5c014790 con 0x7fcf6c0f5410 2024-02-12T16:32:17.553 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.551+0000 7fcf73d63700 1 -- 172.21.15.149:0/2055389644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf6c10c5f0 con 0x7fcf6c0f5410 2024-02-12T16:32:17.554 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.552+0000 7fcf627fc700 1 -- 172.21.15.149:0/2055389644 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcf5c01d380 con 0x7fcf6c0f5410 2024-02-12T16:32:17.554 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:17.554 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:17.556 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.553+0000 7fcf73d63700 1 -- 172.21.15.149:0/2055389644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcf5802ff10 msgr2=0x7fcf580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:17.556 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.553+0000 7fcf73d63700 1 --2- 172.21.15.149:0/2055389644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcf5802ff10 0x7fcf580323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcf680096a0 tx=0x7fcf68006b40).stop 2024-02-12T16:32:17.556 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.553+0000 7fcf73d63700 1 -- 172.21.15.149:0/2055389644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf6c0f5410 msgr2=0x7fcf6c10b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:17.556 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.553+0000 7fcf73d63700 1 --2- 172.21.15.149:0/2055389644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf6c0f5410 0x7fcf6c10b8e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fcf5c005010 tx=0x7fcf5c00b040).stop 2024-02-12T16:32:17.556 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.553+0000 7fcf73d63700 1 -- 172.21.15.149:0/2055389644 shutdown_connections 2024-02-12T16:32:17.556 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.553+0000 7fcf73d63700 1 --2- 172.21.15.149:0/2055389644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcf5802ff10 0x7fcf580323d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:17.556 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.553+0000 7fcf73d63700 1 --2- 172.21.15.149:0/2055389644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf6c0f5410 0x7fcf6c10b8e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:17.556 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.553+0000 7fcf73d63700 1 -- 172.21.15.149:0/2055389644 >> 172.21.15.149:0/2055389644 conn(0x7fcf6c0f0b70 msgr2=0x7fcf6c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:17.556 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.553+0000 7fcf73d63700 1 -- 172.21.15.149:0/2055389644 shutdown_connections 2024-02-12T16:32:17.556 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:17.553+0000 7fcf73d63700 1 -- 172.21.15.149:0/2055389644 wait complete. 2024-02-12T16:32:17.556 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:18.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:16.298458+0000 mgr.smithi104.ohwbhf (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:18.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:17.552755+0000 mon.smithi104 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.149:0/2055389644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:19.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:19.116 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:18.298723+0000 mgr.smithi104.ohwbhf (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:20.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.774+0000 7f256ea1e700 1 -- 172.21.15.149:0/2943599224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2568060120 msgr2=0x7f25680f8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:20.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.774+0000 7f256ea1e700 1 --2- 172.21.15.149:0/2943599224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2568060120 0x7f25680f8fe0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f2564009fb0 tx=0x7f2564009b30).stop 2024-02-12T16:32:20.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.774+0000 7f256ea1e700 1 -- 172.21.15.149:0/2943599224 shutdown_connections 2024-02-12T16:32:20.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.774+0000 7f256ea1e700 1 --2- 172.21.15.149:0/2943599224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2568060120 0x7f25680f8fe0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:20.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.774+0000 7f256ea1e700 1 -- 172.21.15.149:0/2943599224 >> 172.21.15.149:0/2943599224 conn(0x7f25680f09a0 msgr2=0x7f25680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:20.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.774+0000 7f256ea1e700 1 -- 172.21.15.149:0/2943599224 shutdown_connections 2024-02-12T16:32:20.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.774+0000 7f256ea1e700 1 -- 172.21.15.149:0/2943599224 wait complete. 2024-02-12T16:32:20.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.774+0000 7f256ea1e700 1 Processor -- start 2024-02-12T16:32:20.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.775+0000 7f256ea1e700 1 -- start start 2024-02-12T16:32:20.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.775+0000 7f256ea1e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2568060120 0x7f25681065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:20.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.775+0000 7f256ea1e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2568106ab0 con 0x7f2568060120 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.775+0000 7f256da1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2568060120 0x7f25681065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.775+0000 7f256da1c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2568060120 0x7f25681065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:58466/0 (socket says 172.21.15.149:58466) 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.775+0000 7f256da1c700 1 -- 172.21.15.149:0/3285516524 learned_addr learned my addr 172.21.15.149:0/3285516524 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.776+0000 7f256da1c700 1 -- 172.21.15.149:0/3285516524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2564005040 con 0x7f2568060120 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.776+0000 7f256da1c700 1 --2- 172.21.15.149:0/3285516524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2568060120 0x7f25681065c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f2564005010 tx=0x7f256400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.776+0000 7f255e7fc700 1 -- 172.21.15.149:0/3285516524 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2564009320 con 0x7f2568060120 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.776+0000 7f256ea1e700 1 -- 172.21.15.149:0/3285516524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2568106cb0 con 0x7f2568060120 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.776+0000 7f255e7fc700 1 -- 172.21.15.149:0/3285516524 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2564013380 con 0x7f2568060120 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.776+0000 7f256ea1e700 1 -- 172.21.15.149:0/3285516524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2568107140 con 0x7f2568060120 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.776+0000 7f255e7fc700 1 -- 172.21.15.149:0/3285516524 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2564014e50 con 0x7f2568060120 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.777+0000 7f255e7fc700 1 -- 172.21.15.149:0/3285516524 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2564014420 con 0x7f2568060120 2024-02-12T16:32:20.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.777+0000 7f256ea1e700 1 -- 172.21.15.149:0/3285516524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2568045ad0 con 0x7f2568060120 2024-02-12T16:32:20.780 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.777+0000 7f255e7fc700 1 --2- 172.21.15.149:0/3285516524 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f255402ff10 0x7f25540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:20.780 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.777+0000 7f255e7fc700 1 -- 172.21.15.149:0/3285516524 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2564040f70 con 0x7f2568060120 2024-02-12T16:32:20.780 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.777+0000 7f256d21b700 1 --2- 172.21.15.149:0/3285516524 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f255402ff10 0x7f25540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:20.780 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.777+0000 7f256d21b700 1 --2- 172.21.15.149:0/3285516524 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f255402ff10 0x7f25540323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f25580096a0 tx=0x7f2558006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:20.783 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.781+0000 7f255e7fc700 1 -- 172.21.15.149:0/3285516524 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2564014740 con 0x7f2568060120 2024-02-12T16:32:20.990 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.987+0000 7f256ea1e700 1 -- 172.21.15.149:0/3285516524 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2568107f90 con 0x7f2568060120 2024-02-12T16:32:20.990 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.988+0000 7f255e7fc700 1 -- 172.21.15.149:0/3285516524 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2564010230 con 0x7f2568060120 2024-02-12T16:32:20.990 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:20.990 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:20.992 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.990+0000 7f256ea1e700 1 -- 172.21.15.149:0/3285516524 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f255402ff10 msgr2=0x7f25540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:20.992 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.990+0000 7f256ea1e700 1 --2- 172.21.15.149:0/3285516524 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f255402ff10 0x7f25540323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f25580096a0 tx=0x7f2558006b40).stop 2024-02-12T16:32:20.992 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.990+0000 7f256ea1e700 1 -- 172.21.15.149:0/3285516524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2568060120 msgr2=0x7f25681065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:20.992 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.990+0000 7f256ea1e700 1 --2- 172.21.15.149:0/3285516524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2568060120 0x7f25681065c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f2564005010 tx=0x7f256400b040).stop 2024-02-12T16:32:20.992 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.990+0000 7f256ea1e700 1 -- 172.21.15.149:0/3285516524 shutdown_connections 2024-02-12T16:32:20.992 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.990+0000 7f256ea1e700 1 --2- 172.21.15.149:0/3285516524 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f255402ff10 0x7f25540323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:20.992 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.990+0000 7f256ea1e700 1 --2- 172.21.15.149:0/3285516524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2568060120 0x7f25681065c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:20.992 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.990+0000 7f256ea1e700 1 -- 172.21.15.149:0/3285516524 >> 172.21.15.149:0/3285516524 conn(0x7f25680f09a0 msgr2=0x7f25680f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:20.992 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.990+0000 7f256ea1e700 1 -- 172.21.15.149:0/3285516524 shutdown_connections 2024-02-12T16:32:20.992 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:20.990+0000 7f256ea1e700 1 -- 172.21.15.149:0/3285516524 wait complete. 2024-02-12T16:32:20.993 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:21.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:21 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:20.989510+0000 mon.smithi104 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.149:0/3285516524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:22.645 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:22.646 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:22.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:20.299015+0000 mgr.smithi104.ohwbhf (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:24.577 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.573+0000 7f518027e700 1 -- 172.21.15.149:0/1914133685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51780f5410 msgr2=0x7f51780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:24.577 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.573+0000 7f518027e700 1 --2- 172.21.15.149:0/1914133685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51780f5410 0x7f51780f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f5170009fb0 tx=0x7f5170009b30).stop 2024-02-12T16:32:24.577 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.573+0000 7f518027e700 1 -- 172.21.15.149:0/1914133685 shutdown_connections 2024-02-12T16:32:24.577 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.573+0000 7f518027e700 1 --2- 172.21.15.149:0/1914133685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51780f5410 0x7f51780f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:24.577 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.573+0000 7f518027e700 1 -- 172.21.15.149:0/1914133685 >> 172.21.15.149:0/1914133685 conn(0x7f51780f0b70 msgr2=0x7f51780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:24.577 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.573+0000 7f518027e700 1 -- 172.21.15.149:0/1914133685 shutdown_connections 2024-02-12T16:32:24.577 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.573+0000 7f518027e700 1 -- 172.21.15.149:0/1914133685 wait complete. 2024-02-12T16:32:24.577 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.574+0000 7f518027e700 1 Processor -- start 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.574+0000 7f518027e700 1 -- start start 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.574+0000 7f518027e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51780f5410 0x7f51781067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.574+0000 7f518027e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5178106cd0 con 0x7f51780f5410 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.574+0000 7f517e01a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51780f5410 0x7f51781067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.574+0000 7f517e01a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51780f5410 0x7f51781067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:60324/0 (socket says 172.21.15.149:60324) 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.574+0000 7f517e01a700 1 -- 172.21.15.149:0/955010488 learned_addr learned my addr 172.21.15.149:0/955010488 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.574+0000 7f517e01a700 1 -- 172.21.15.149:0/955010488 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5170005040 con 0x7f51780f5410 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.574+0000 7f517e01a700 1 --2- 172.21.15.149:0/955010488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51780f5410 0x7f51781067e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f5170005010 tx=0x7f517000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.575+0000 7f5176ffd700 1 -- 172.21.15.149:0/955010488 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5170009320 con 0x7f51780f5410 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.575+0000 7f518027e700 1 -- 172.21.15.149:0/955010488 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5178106ed0 con 0x7f51780f5410 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.575+0000 7f5176ffd700 1 -- 172.21.15.149:0/955010488 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5170013380 con 0x7f51780f5410 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.575+0000 7f5176ffd700 1 -- 172.21.15.149:0/955010488 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5170014e50 con 0x7f51780f5410 2024-02-12T16:32:24.578 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.575+0000 7f518027e700 1 -- 172.21.15.149:0/955010488 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5178107360 con 0x7f51780f5410 2024-02-12T16:32:24.579 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.575+0000 7f5176ffd700 1 -- 172.21.15.149:0/955010488 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51700095f0 con 0x7f51780f5410 2024-02-12T16:32:24.579 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.576+0000 7f5176ffd700 1 --2- 172.21.15.149:0/955010488 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f516002ff10 0x7f51600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:24.579 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.576+0000 7f5176ffd700 1 -- 172.21.15.149:0/955010488 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5170040fa0 con 0x7f51780f5410 2024-02-12T16:32:24.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.576+0000 7f518027e700 1 -- 172.21.15.149:0/955010488 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5178059100 con 0x7f51780f5410 2024-02-12T16:32:24.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.576+0000 7f517d819700 1 --2- 172.21.15.149:0/955010488 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f516002ff10 0x7f51600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:24.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.576+0000 7f517d819700 1 --2- 172.21.15.149:0/955010488 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f516002ff10 0x7f51600323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f516c0096a0 tx=0x7f516c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:24.583 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.581+0000 7f5176ffd700 1 -- 172.21.15.149:0/955010488 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5170014420 con 0x7f51780f5410 2024-02-12T16:32:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:22.299370+0000 mgr.smithi104.ohwbhf (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:24.787 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.785+0000 7f518027e700 1 -- 172.21.15.149:0/955010488 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51780fc0d0 con 0x7f51780f5410 2024-02-12T16:32:24.788 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.786+0000 7f5176ffd700 1 -- 172.21.15.149:0/955010488 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5170010230 con 0x7f51780f5410 2024-02-12T16:32:24.788 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:24.788 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:24.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.787+0000 7f518027e700 1 -- 172.21.15.149:0/955010488 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f516002ff10 msgr2=0x7f51600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:24.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.787+0000 7f518027e700 1 --2- 172.21.15.149:0/955010488 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f516002ff10 0x7f51600323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f516c0096a0 tx=0x7f516c006b40).stop 2024-02-12T16:32:24.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.787+0000 7f518027e700 1 -- 172.21.15.149:0/955010488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51780f5410 msgr2=0x7f51781067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:24.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.787+0000 7f518027e700 1 --2- 172.21.15.149:0/955010488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51780f5410 0x7f51781067e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f5170005010 tx=0x7f517000b040).stop 2024-02-12T16:32:24.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.788+0000 7f518027e700 1 -- 172.21.15.149:0/955010488 shutdown_connections 2024-02-12T16:32:24.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.788+0000 7f518027e700 1 --2- 172.21.15.149:0/955010488 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f516002ff10 0x7f51600323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:24.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.788+0000 7f518027e700 1 --2- 172.21.15.149:0/955010488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f51780f5410 0x7f51781067e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:24.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.788+0000 7f518027e700 1 -- 172.21.15.149:0/955010488 >> 172.21.15.149:0/955010488 conn(0x7f51780f0b70 msgr2=0x7f51780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:24.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.788+0000 7f518027e700 1 -- 172.21.15.149:0/955010488 shutdown_connections 2024-02-12T16:32:24.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:24.788+0000 7f518027e700 1 -- 172.21.15.149:0/955010488 wait complete. 2024-02-12T16:32:24.791 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:24.404354+0000 mon.smithi104 (mon.0) 159 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:32:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:24.411397+0000 mon.smithi104 (mon.0) 160 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:32:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:24.787096+0000 mon.smithi104 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.149:0/955010488' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:26.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:26.449 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:26.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:24.299612+0000 mgr.smithi104.ohwbhf (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:28.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.191+0000 7ffbde96d700 1 -- 172.21.15.149:0/64168296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbd80f51a0 msgr2=0x7ffbd80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:28.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.191+0000 7ffbde96d700 1 --2- 172.21.15.149:0/64168296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbd80f51a0 0x7ffbd80f55c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ffbd4009fb0 tx=0x7ffbd4009b30).stop 2024-02-12T16:32:28.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.191+0000 7ffbde96d700 1 -- 172.21.15.149:0/64168296 shutdown_connections 2024-02-12T16:32:28.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.191+0000 7ffbde96d700 1 --2- 172.21.15.149:0/64168296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbd80f51a0 0x7ffbd80f55c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:28.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.191+0000 7ffbde96d700 1 -- 172.21.15.149:0/64168296 >> 172.21.15.149:0/64168296 conn(0x7ffbd80f09a0 msgr2=0x7ffbd80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:28.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.191+0000 7ffbde96d700 1 -- 172.21.15.149:0/64168296 shutdown_connections 2024-02-12T16:32:28.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.191+0000 7ffbde96d700 1 -- 172.21.15.149:0/64168296 wait complete. 2024-02-12T16:32:28.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.191+0000 7ffbde96d700 1 Processor -- start 2024-02-12T16:32:28.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.191+0000 7ffbde96d700 1 -- start start 2024-02-12T16:32:28.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbde96d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbd80f51a0 0x7ffbd81087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:28.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbde96d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd8108cc0 con 0x7ffbd80f51a0 2024-02-12T16:32:28.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbdd96b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbd80f51a0 0x7ffbd81087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:28.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbdd96b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbd80f51a0 0x7ffbd81087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:60340/0 (socket says 172.21.15.149:60340) 2024-02-12T16:32:28.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbdd96b700 1 -- 172.21.15.149:0/1470057803 learned_addr learned my addr 172.21.15.149:0/1470057803 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:28.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbdd96b700 1 -- 172.21.15.149:0/1470057803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd4005040 con 0x7ffbd80f51a0 2024-02-12T16:32:28.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbdd96b700 1 --2- 172.21.15.149:0/1470057803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbd80f51a0 0x7ffbd81087d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ffbd40089f0 tx=0x7ffbd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:28.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbce7fc700 1 -- 172.21.15.149:0/1470057803 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbd4009320 con 0x7ffbd80f51a0 2024-02-12T16:32:28.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbde96d700 1 -- 172.21.15.149:0/1470057803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd8108ec0 con 0x7ffbd80f51a0 2024-02-12T16:32:28.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbce7fc700 1 -- 172.21.15.149:0/1470057803 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbd4013380 con 0x7ffbd80f51a0 2024-02-12T16:32:28.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbce7fc700 1 -- 172.21.15.149:0/1470057803 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbd4014e20 con 0x7ffbd80f51a0 2024-02-12T16:32:28.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.192+0000 7ffbde96d700 1 -- 172.21.15.149:0/1470057803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd8109350 con 0x7ffbd80f51a0 2024-02-12T16:32:28.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.193+0000 7ffbce7fc700 1 -- 172.21.15.149:0/1470057803 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffbd4014420 con 0x7ffbd80f51a0 2024-02-12T16:32:28.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.193+0000 7ffbde96d700 1 -- 172.21.15.149:0/1470057803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd8045ad0 con 0x7ffbd80f51a0 2024-02-12T16:32:28.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.193+0000 7ffbce7fc700 1 --2- 172.21.15.149:0/1470057803 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffbc402ff60 0x7ffbc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:28.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.193+0000 7ffbce7fc700 1 -- 172.21.15.149:0/1470057803 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffbd4040ee0 con 0x7ffbd80f51a0 2024-02-12T16:32:28.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.194+0000 7ffbdd16a700 1 --2- 172.21.15.149:0/1470057803 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffbc402ff60 0x7ffbc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:28.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.194+0000 7ffbdd16a700 1 --2- 172.21.15.149:0/1470057803 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffbc402ff60 0x7ffbc4032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ffbc80096a0 tx=0x7ffbc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:28.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.198+0000 7ffbce7fc700 1 -- 172.21.15.149:0/1470057803 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffbd4014790 con 0x7ffbd80f51a0 2024-02-12T16:32:28.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.403+0000 7ffbde96d700 1 -- 172.21.15.149:0/1470057803 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbd810a200 con 0x7ffbd80f51a0 2024-02-12T16:32:28.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.404+0000 7ffbce7fc700 1 -- 172.21.15.149:0/1470057803 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffbd4010230 con 0x7ffbd80f51a0 2024-02-12T16:32:28.406 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:28.406 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:28.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.405+0000 7ffbde96d700 1 -- 172.21.15.149:0/1470057803 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffbc402ff60 msgr2=0x7ffbc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:28.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.405+0000 7ffbde96d700 1 --2- 172.21.15.149:0/1470057803 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffbc402ff60 0x7ffbc4032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ffbc80096a0 tx=0x7ffbc8006b40).stop 2024-02-12T16:32:28.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.405+0000 7ffbde96d700 1 -- 172.21.15.149:0/1470057803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbd80f51a0 msgr2=0x7ffbd81087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:28.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.405+0000 7ffbde96d700 1 --2- 172.21.15.149:0/1470057803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbd80f51a0 0x7ffbd81087d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ffbd40089f0 tx=0x7ffbd400b040).stop 2024-02-12T16:32:28.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.405+0000 7ffbde96d700 1 -- 172.21.15.149:0/1470057803 shutdown_connections 2024-02-12T16:32:28.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.405+0000 7ffbde96d700 1 --2- 172.21.15.149:0/1470057803 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffbc402ff60 0x7ffbc4032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:28.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.405+0000 7ffbde96d700 1 --2- 172.21.15.149:0/1470057803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffbd80f51a0 0x7ffbd81087d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:28.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.405+0000 7ffbde96d700 1 -- 172.21.15.149:0/1470057803 >> 172.21.15.149:0/1470057803 conn(0x7ffbd80f09a0 msgr2=0x7ffbd805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:28.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.405+0000 7ffbde96d700 1 -- 172.21.15.149:0/1470057803 shutdown_connections 2024-02-12T16:32:28.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:28.405+0000 7ffbde96d700 1 -- 172.21.15.149:0/1470057803 wait complete. 2024-02-12T16:32:28.408 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:28.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:26.299948+0000 mgr.smithi104.ohwbhf (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:29.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:29 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:28.404965+0000 mon.smithi104 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.149:0/1470057803' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:30.050 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:30.050 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:30.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:28.300207+0000 mgr.smithi104.ohwbhf (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:31.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.959+0000 7fad99cd9700 1 -- 172.21.15.149:0/1566864051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad940f6510 msgr2=0x7fad940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:31.961 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.959+0000 7fad99cd9700 1 --2- 172.21.15.149:0/1566864051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad940f6510 0x7fad940f6930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fad88009fb0 tx=0x7fad88009b30).stop 2024-02-12T16:32:31.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.959+0000 7fad99cd9700 1 -- 172.21.15.149:0/1566864051 shutdown_connections 2024-02-12T16:32:31.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.959+0000 7fad99cd9700 1 --2- 172.21.15.149:0/1566864051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad940f6510 0x7fad940f6930 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:31.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.959+0000 7fad99cd9700 1 -- 172.21.15.149:0/1566864051 >> 172.21.15.149:0/1566864051 conn(0x7fad940f1c70 msgr2=0x7fad940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:31.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.959+0000 7fad99cd9700 1 -- 172.21.15.149:0/1566864051 shutdown_connections 2024-02-12T16:32:31.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.959+0000 7fad99cd9700 1 -- 172.21.15.149:0/1566864051 wait complete. 2024-02-12T16:32:31.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.960+0000 7fad99cd9700 1 Processor -- start 2024-02-12T16:32:31.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.960+0000 7fad99cd9700 1 -- start start 2024-02-12T16:32:31.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.960+0000 7fad99cd9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad940f6510 0x7fad941067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:31.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.960+0000 7fad99cd9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad94106cd0 con 0x7fad940f6510 2024-02-12T16:32:31.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.961+0000 7fad937fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad940f6510 0x7fad941067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:31.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.961+0000 7fad937fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad940f6510 0x7fad941067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:60358/0 (socket says 172.21.15.149:60358) 2024-02-12T16:32:31.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.961+0000 7fad937fe700 1 -- 172.21.15.149:0/1198377578 learned_addr learned my addr 172.21.15.149:0/1198377578 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:31.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.961+0000 7fad937fe700 1 -- 172.21.15.149:0/1198377578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad88005040 con 0x7fad940f6510 2024-02-12T16:32:31.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.961+0000 7fad937fe700 1 --2- 172.21.15.149:0/1198377578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad940f6510 0x7fad941067e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fad88005010 tx=0x7fad8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:31.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.961+0000 7fad7bfff700 1 -- 172.21.15.149:0/1198377578 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad88009320 con 0x7fad940f6510 2024-02-12T16:32:31.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.961+0000 7fad7bfff700 1 -- 172.21.15.149:0/1198377578 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad88013380 con 0x7fad940f6510 2024-02-12T16:32:31.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.961+0000 7fad99cd9700 1 -- 172.21.15.149:0/1198377578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad94106ed0 con 0x7fad940f6510 2024-02-12T16:32:31.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.961+0000 7fad7bfff700 1 -- 172.21.15.149:0/1198377578 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad88014e50 con 0x7fad940f6510 2024-02-12T16:32:31.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.961+0000 7fad99cd9700 1 -- 172.21.15.149:0/1198377578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad94107360 con 0x7fad940f6510 2024-02-12T16:32:31.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.962+0000 7fad7bfff700 1 -- 172.21.15.149:0/1198377578 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad88014420 con 0x7fad940f6510 2024-02-12T16:32:31.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.962+0000 7fad99cd9700 1 -- 172.21.15.149:0/1198377578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad94045ad0 con 0x7fad940f6510 2024-02-12T16:32:31.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.962+0000 7fad7bfff700 1 --2- 172.21.15.149:0/1198377578 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fad7c02ff10 0x7fad7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:31.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.962+0000 7fad7bfff700 1 -- 172.21.15.149:0/1198377578 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad88040e30 con 0x7fad940f6510 2024-02-12T16:32:31.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.963+0000 7fad92ffd700 1 --2- 172.21.15.149:0/1198377578 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fad7c02ff10 0x7fad7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:31.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.963+0000 7fad92ffd700 1 --2- 172.21.15.149:0/1198377578 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fad7c02ff10 0x7fad7c0323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fad840096a0 tx=0x7fad84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:31.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:31.967+0000 7fad7bfff700 1 -- 172.21.15.149:0/1198377578 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad880095f0 con 0x7fad940f6510 2024-02-12T16:32:32.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.172+0000 7fad99cd9700 1 -- 172.21.15.149:0/1198377578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad94108210 con 0x7fad940f6510 2024-02-12T16:32:32.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.172+0000 7fad7bfff700 1 -- 172.21.15.149:0/1198377578 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fad8801db70 con 0x7fad940f6510 2024-02-12T16:32:32.175 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:32.175 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:32.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.174+0000 7fad99cd9700 1 -- 172.21.15.149:0/1198377578 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fad7c02ff10 msgr2=0x7fad7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:32.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.174+0000 7fad99cd9700 1 --2- 172.21.15.149:0/1198377578 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fad7c02ff10 0x7fad7c0323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fad840096a0 tx=0x7fad84006b40).stop 2024-02-12T16:32:32.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.174+0000 7fad99cd9700 1 -- 172.21.15.149:0/1198377578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad940f6510 msgr2=0x7fad941067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:32.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.174+0000 7fad99cd9700 1 --2- 172.21.15.149:0/1198377578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad940f6510 0x7fad941067e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fad88005010 tx=0x7fad8800b040).stop 2024-02-12T16:32:32.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.174+0000 7fad99cd9700 1 -- 172.21.15.149:0/1198377578 shutdown_connections 2024-02-12T16:32:32.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.174+0000 7fad99cd9700 1 --2- 172.21.15.149:0/1198377578 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fad7c02ff10 0x7fad7c0323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:32.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.174+0000 7fad99cd9700 1 --2- 172.21.15.149:0/1198377578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fad940f6510 0x7fad941067e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:32.177 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.174+0000 7fad99cd9700 1 -- 172.21.15.149:0/1198377578 >> 172.21.15.149:0/1198377578 conn(0x7fad940f1c70 msgr2=0x7fad940f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:32.177 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.174+0000 7fad99cd9700 1 -- 172.21.15.149:0/1198377578 shutdown_connections 2024-02-12T16:32:32.177 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:32.174+0000 7fad99cd9700 1 -- 172.21.15.149:0/1198377578 wait complete. 2024-02-12T16:32:32.177 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:32.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:30.300540+0000 mgr.smithi104.ohwbhf (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:32.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:32.173557+0000 mon.smithi104 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.149:0/1198377578' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:33.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:33.771 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:34.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:32.300900+0000 mgr.smithi104.ohwbhf (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:35.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.472+0000 7fc721059700 1 -- 172.21.15.149:0/193815066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc71c0f5410 msgr2=0x7fc71c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:35.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.472+0000 7fc721059700 1 --2- 172.21.15.149:0/193815066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc71c0f5410 0x7fc71c0f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc710009fb0 tx=0x7fc710009b30).stop 2024-02-12T16:32:35.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.472+0000 7fc721059700 1 -- 172.21.15.149:0/193815066 shutdown_connections 2024-02-12T16:32:35.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.472+0000 7fc721059700 1 --2- 172.21.15.149:0/193815066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc71c0f5410 0x7fc71c0f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:35.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.472+0000 7fc721059700 1 -- 172.21.15.149:0/193815066 >> 172.21.15.149:0/193815066 conn(0x7fc71c0f0b70 msgr2=0x7fc71c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:35.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.472+0000 7fc721059700 1 -- 172.21.15.149:0/193815066 shutdown_connections 2024-02-12T16:32:35.476 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.472+0000 7fc721059700 1 -- 172.21.15.149:0/193815066 wait complete. 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.473+0000 7fc721059700 1 Processor -- start 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.473+0000 7fc721059700 1 -- start start 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.473+0000 7fc721059700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc71c0f5410 0x7fc71c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.473+0000 7fc721059700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c106cd0 con 0x7fc71c0f5410 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.473+0000 7fc71ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc71c0f5410 0x7fc71c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.473+0000 7fc71ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc71c0f5410 0x7fc71c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:59680/0 (socket says 172.21.15.149:59680) 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.473+0000 7fc71ad9d700 1 -- 172.21.15.149:0/3940908258 learned_addr learned my addr 172.21.15.149:0/3940908258 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.474+0000 7fc71ad9d700 1 -- 172.21.15.149:0/3940908258 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc710005040 con 0x7fc71c0f5410 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.474+0000 7fc71ad9d700 1 --2- 172.21.15.149:0/3940908258 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc71c0f5410 0x7fc71c1067e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fc710005010 tx=0x7fc71000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.474+0000 7fc7037fe700 1 -- 172.21.15.149:0/3940908258 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc710009320 con 0x7fc71c0f5410 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.474+0000 7fc721059700 1 -- 172.21.15.149:0/3940908258 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71c106ed0 con 0x7fc71c0f5410 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.474+0000 7fc7037fe700 1 -- 172.21.15.149:0/3940908258 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc710013380 con 0x7fc71c0f5410 2024-02-12T16:32:35.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.474+0000 7fc7037fe700 1 -- 172.21.15.149:0/3940908258 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc710014e50 con 0x7fc71c0f5410 2024-02-12T16:32:35.478 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.474+0000 7fc721059700 1 -- 172.21.15.149:0/3940908258 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71c107360 con 0x7fc71c0f5410 2024-02-12T16:32:35.478 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.475+0000 7fc7037fe700 1 -- 172.21.15.149:0/3940908258 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc710014420 con 0x7fc71c0f5410 2024-02-12T16:32:35.479 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.475+0000 7fc721059700 1 -- 172.21.15.149:0/3940908258 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71c045ad0 con 0x7fc71c0f5410 2024-02-12T16:32:35.479 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.475+0000 7fc7037fe700 1 --2- 172.21.15.149:0/3940908258 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc70402ff10 0x7fc7040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:35.479 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.475+0000 7fc7037fe700 1 -- 172.21.15.149:0/3940908258 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc710040f70 con 0x7fc71c0f5410 2024-02-12T16:32:35.479 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.475+0000 7fc71a59c700 1 --2- 172.21.15.149:0/3940908258 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc70402ff10 0x7fc7040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:35.479 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.475+0000 7fc71a59c700 1 --2- 172.21.15.149:0/3940908258 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc70402ff10 0x7fc7040323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc70c0096a0 tx=0x7fc70c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:35.481 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.479+0000 7fc7037fe700 1 -- 172.21.15.149:0/3940908258 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7100095f0 con 0x7fc71c0f5410 2024-02-12T16:32:35.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.684+0000 7fc721059700 1 -- 172.21.15.149:0/3940908258 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc71c1081b0 con 0x7fc71c0f5410 2024-02-12T16:32:35.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.684+0000 7fc7037fe700 1 -- 172.21.15.149:0/3940908258 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc71001db70 con 0x7fc71c0f5410 2024-02-12T16:32:35.686 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:35.687 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:35.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.686+0000 7fc721059700 1 -- 172.21.15.149:0/3940908258 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc70402ff10 msgr2=0x7fc7040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:35.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.686+0000 7fc721059700 1 --2- 172.21.15.149:0/3940908258 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc70402ff10 0x7fc7040323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc70c0096a0 tx=0x7fc70c006b40).stop 2024-02-12T16:32:35.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.686+0000 7fc721059700 1 -- 172.21.15.149:0/3940908258 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc71c0f5410 msgr2=0x7fc71c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:35.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.686+0000 7fc721059700 1 --2- 172.21.15.149:0/3940908258 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc71c0f5410 0x7fc71c1067e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fc710005010 tx=0x7fc71000b040).stop 2024-02-12T16:32:35.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.686+0000 7fc721059700 1 -- 172.21.15.149:0/3940908258 shutdown_connections 2024-02-12T16:32:35.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.686+0000 7fc721059700 1 --2- 172.21.15.149:0/3940908258 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc70402ff10 0x7fc7040323d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:35.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.686+0000 7fc721059700 1 --2- 172.21.15.149:0/3940908258 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc71c0f5410 0x7fc71c1067e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:35.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.686+0000 7fc721059700 1 -- 172.21.15.149:0/3940908258 >> 172.21.15.149:0/3940908258 conn(0x7fc71c0f0b70 msgr2=0x7fc71c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:35.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.686+0000 7fc721059700 1 -- 172.21.15.149:0/3940908258 shutdown_connections 2024-02-12T16:32:35.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:35.686+0000 7fc721059700 1 -- 172.21.15.149:0/3940908258 wait complete. 2024-02-12T16:32:35.689 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:36.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:34.301224+0000 mgr.smithi104.ohwbhf (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:36.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:35.685672+0000 mon.smithi104 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.149:0/3940908258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:37.291 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:37.292 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:38.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:36.301554+0000 mgr.smithi104.ohwbhf (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:38.981 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.965+0000 7f36110f7700 1 -- 172.21.15.149:0/628544397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f360c0f5410 msgr2=0x7f360c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:38.981 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.965+0000 7f36110f7700 1 --2- 172.21.15.149:0/628544397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f360c0f5410 0x7f360c0f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3600009fb0 tx=0x7f3600009b30).stop 2024-02-12T16:32:38.981 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.965+0000 7f36110f7700 1 -- 172.21.15.149:0/628544397 shutdown_connections 2024-02-12T16:32:38.981 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.965+0000 7f36110f7700 1 --2- 172.21.15.149:0/628544397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f360c0f5410 0x7f360c0f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.965+0000 7f36110f7700 1 -- 172.21.15.149:0/628544397 >> 172.21.15.149:0/628544397 conn(0x7f360c0f0b70 msgr2=0x7f360c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.966+0000 7f36110f7700 1 -- 172.21.15.149:0/628544397 shutdown_connections 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.966+0000 7f36110f7700 1 -- 172.21.15.149:0/628544397 wait complete. 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.966+0000 7f36110f7700 1 Processor -- start 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.966+0000 7f36110f7700 1 -- start start 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.966+0000 7f36110f7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f360c0f5410 0x7f360c10b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.966+0000 7f36110f7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c107330 con 0x7f360c0f5410 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.967+0000 7f360ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f360c0f5410 0x7f360c10b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.967+0000 7f360ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f360c0f5410 0x7f360c10b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:59710/0 (socket says 172.21.15.149:59710) 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.967+0000 7f360ad9d700 1 -- 172.21.15.149:0/1688348827 learned_addr learned my addr 172.21.15.149:0/1688348827 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.967+0000 7f360ad9d700 1 -- 172.21.15.149:0/1688348827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3600005040 con 0x7f360c0f5410 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.967+0000 7f360ad9d700 1 --2- 172.21.15.149:0/1688348827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f360c0f5410 0x7f360c10b940 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f36000089f0 tx=0x7f360000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:38.982 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.967+0000 7f35f37fe700 1 -- 172.21.15.149:0/1688348827 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3600009320 con 0x7f360c0f5410 2024-02-12T16:32:38.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.967+0000 7f36110f7700 1 -- 172.21.15.149:0/1688348827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360c060220 con 0x7f360c0f5410 2024-02-12T16:32:38.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.967+0000 7f35f37fe700 1 -- 172.21.15.149:0/1688348827 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3600013380 con 0x7f360c0f5410 2024-02-12T16:32:38.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.967+0000 7f35f37fe700 1 -- 172.21.15.149:0/1688348827 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f360001ed80 con 0x7f360c0f5410 2024-02-12T16:32:38.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.967+0000 7f36110f7700 1 -- 172.21.15.149:0/1688348827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f360c10be30 con 0x7f360c0f5410 2024-02-12T16:32:38.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.968+0000 7f35f37fe700 1 -- 172.21.15.149:0/1688348827 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3600014420 con 0x7f360c0f5410 2024-02-12T16:32:38.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.968+0000 7f35f37fe700 1 --2- 172.21.15.149:0/1688348827 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f35f402ff60 0x7f35f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:38.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.968+0000 7f35f37fe700 1 -- 172.21.15.149:0/1688348827 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3600040f10 con 0x7f360c0f5410 2024-02-12T16:32:38.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.968+0000 7f36110f7700 1 -- 172.21.15.149:0/1688348827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f360c045ad0 con 0x7f360c0f5410 2024-02-12T16:32:38.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.968+0000 7f360a59c700 1 --2- 172.21.15.149:0/1688348827 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f35f402ff60 0x7f35f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:38.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.969+0000 7f360a59c700 1 --2- 172.21.15.149:0/1688348827 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f35f402ff60 0x7f35f4032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f35fc0096a0 tx=0x7f35fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:38.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:38.973+0000 7f35f37fe700 1 -- 172.21.15.149:0/1688348827 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f360001c910 con 0x7f360c0f5410 2024-02-12T16:32:39.177 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.176+0000 7f36110f7700 1 -- 172.21.15.149:0/1688348827 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f360c10c580 con 0x7f360c0f5410 2024-02-12T16:32:39.177 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.176+0000 7f35f37fe700 1 -- 172.21.15.149:0/1688348827 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f360001c320 con 0x7f360c0f5410 2024-02-12T16:32:39.178 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:39.178 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:39.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.178+0000 7f36110f7700 1 -- 172.21.15.149:0/1688348827 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f35f402ff60 msgr2=0x7f35f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:39.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.178+0000 7f36110f7700 1 --2- 172.21.15.149:0/1688348827 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f35f402ff60 0x7f35f4032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f35fc0096a0 tx=0x7f35fc006b40).stop 2024-02-12T16:32:39.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.178+0000 7f36110f7700 1 -- 172.21.15.149:0/1688348827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f360c0f5410 msgr2=0x7f360c10b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:39.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.178+0000 7f36110f7700 1 --2- 172.21.15.149:0/1688348827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f360c0f5410 0x7f360c10b940 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f36000089f0 tx=0x7f360000b040).stop 2024-02-12T16:32:39.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.178+0000 7f36110f7700 1 -- 172.21.15.149:0/1688348827 shutdown_connections 2024-02-12T16:32:39.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.178+0000 7f36110f7700 1 --2- 172.21.15.149:0/1688348827 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f35f402ff60 0x7f35f4032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:39.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.178+0000 7f36110f7700 1 --2- 172.21.15.149:0/1688348827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f360c0f5410 0x7f360c10b940 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:39.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.178+0000 7f36110f7700 1 -- 172.21.15.149:0/1688348827 >> 172.21.15.149:0/1688348827 conn(0x7f360c0f0b70 msgr2=0x7f360c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:39.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.178+0000 7f36110f7700 1 -- 172.21.15.149:0/1688348827 shutdown_connections 2024-02-12T16:32:39.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:39.178+0000 7f36110f7700 1 -- 172.21.15.149:0/1688348827 wait complete. 2024-02-12T16:32:39.180 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:39.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:39 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:39.176704+0000 mon.smithi104 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.149:0/1688348827' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:40.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:38.301887+0000 mgr.smithi104.ohwbhf (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:40.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:40.763 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:42.617 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.614+0000 7fdcfdc49700 1 -- 172.21.15.149:0/705383477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80f51a0 msgr2=0x7fdcf80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:42.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.615+0000 7fdcfdc49700 1 --2- 172.21.15.149:0/705383477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80f51a0 0x7fdcf80f55c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fdcec009fb0 tx=0x7fdcec009b30).stop 2024-02-12T16:32:42.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.615+0000 7fdcfdc49700 1 -- 172.21.15.149:0/705383477 shutdown_connections 2024-02-12T16:32:42.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.615+0000 7fdcfdc49700 1 --2- 172.21.15.149:0/705383477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80f51a0 0x7fdcf80f55c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:42.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.615+0000 7fdcfdc49700 1 -- 172.21.15.149:0/705383477 >> 172.21.15.149:0/705383477 conn(0x7fdcf80f09a0 msgr2=0x7fdcf80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:42.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.615+0000 7fdcfdc49700 1 -- 172.21.15.149:0/705383477 shutdown_connections 2024-02-12T16:32:42.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.615+0000 7fdcfdc49700 1 -- 172.21.15.149:0/705383477 wait complete. 2024-02-12T16:32:42.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.615+0000 7fdcfdc49700 1 Processor -- start 2024-02-12T16:32:42.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.615+0000 7fdcfdc49700 1 -- start start 2024-02-12T16:32:42.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.615+0000 7fdcfdc49700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80f51a0 0x7fdcf81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:42.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.615+0000 7fdcfdc49700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf8106ac0 con 0x7fdcf80f51a0 2024-02-12T16:32:42.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.616+0000 7fdcfcc47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80f51a0 0x7fdcf81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:42.619 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.616+0000 7fdcfcc47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80f51a0 0x7fdcf81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:59720/0 (socket says 172.21.15.149:59720) 2024-02-12T16:32:42.619 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.616+0000 7fdcfcc47700 1 -- 172.21.15.149:0/1929357656 learned_addr learned my addr 172.21.15.149:0/1929357656 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:42.619 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.616+0000 7fdcfcc47700 1 -- 172.21.15.149:0/1929357656 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec005040 con 0x7fdcf80f51a0 2024-02-12T16:32:42.619 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.616+0000 7fdcfcc47700 1 --2- 172.21.15.149:0/1929357656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80f51a0 0x7fdcf81065d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fdcec0089f0 tx=0x7fdcec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:42.619 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.616+0000 7fdcf57fa700 1 -- 172.21.15.149:0/1929357656 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcec009320 con 0x7fdcf80f51a0 2024-02-12T16:32:42.619 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.616+0000 7fdcf57fa700 1 -- 172.21.15.149:0/1929357656 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcec013380 con 0x7fdcf80f51a0 2024-02-12T16:32:42.619 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.616+0000 7fdcfdc49700 1 -- 172.21.15.149:0/1929357656 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf8106cc0 con 0x7fdcf80f51a0 2024-02-12T16:32:42.619 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.616+0000 7fdcf57fa700 1 -- 172.21.15.149:0/1929357656 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcec014e20 con 0x7fdcf80f51a0 2024-02-12T16:32:42.619 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.616+0000 7fdcfdc49700 1 -- 172.21.15.149:0/1929357656 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf8107150 con 0x7fdcf80f51a0 2024-02-12T16:32:42.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.617+0000 7fdcf57fa700 1 -- 172.21.15.149:0/1929357656 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcec0263a0 con 0x7fdcf80f51a0 2024-02-12T16:32:42.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.617+0000 7fdcfdc49700 1 -- 172.21.15.149:0/1929357656 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf8045ad0 con 0x7fdcf80f51a0 2024-02-12T16:32:42.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.617+0000 7fdcf57fa700 1 --2- 172.21.15.149:0/1929357656 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdce802ff60 0x7fdce8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:42.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.617+0000 7fdcf57fa700 1 -- 172.21.15.149:0/1929357656 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcec0411e0 con 0x7fdcf80f51a0 2024-02-12T16:32:42.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.617+0000 7fdcf7fff700 1 --2- 172.21.15.149:0/1929357656 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdce802ff60 0x7fdce8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:42.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.618+0000 7fdcf7fff700 1 --2- 172.21.15.149:0/1929357656 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdce802ff60 0x7fdce8032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fdce40096a0 tx=0x7fdce4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:42.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.622+0000 7fdcf57fa700 1 -- 172.21.15.149:0/1929357656 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcec01ccf0 con 0x7fdcf80f51a0 2024-02-12T16:32:42.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:40.302260+0000 mgr.smithi104.ohwbhf (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:42.827 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.826+0000 7fdcfdc49700 1 -- 172.21.15.149:0/1929357656 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcf80fad30 con 0x7fdcf80f51a0 2024-02-12T16:32:42.827 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.827+0000 7fdcf57fa700 1 -- 172.21.15.149:0/1929357656 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdcec01cb10 con 0x7fdcf80f51a0 2024-02-12T16:32:42.828 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:42.828 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:42.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.828+0000 7fdcfdc49700 1 -- 172.21.15.149:0/1929357656 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdce802ff60 msgr2=0x7fdce8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:42.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.828+0000 7fdcfdc49700 1 --2- 172.21.15.149:0/1929357656 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdce802ff60 0x7fdce8032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fdce40096a0 tx=0x7fdce4006b40).stop 2024-02-12T16:32:42.830 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.828+0000 7fdcfdc49700 1 -- 172.21.15.149:0/1929357656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80f51a0 msgr2=0x7fdcf81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:42.830 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.828+0000 7fdcfdc49700 1 --2- 172.21.15.149:0/1929357656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80f51a0 0x7fdcf81065d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fdcec0089f0 tx=0x7fdcec00b040).stop 2024-02-12T16:32:42.830 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.828+0000 7fdcfdc49700 1 -- 172.21.15.149:0/1929357656 shutdown_connections 2024-02-12T16:32:42.830 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.828+0000 7fdcfdc49700 1 --2- 172.21.15.149:0/1929357656 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdce802ff60 0x7fdce8032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:42.830 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.828+0000 7fdcfdc49700 1 --2- 172.21.15.149:0/1929357656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80f51a0 0x7fdcf81065d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:42.830 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.828+0000 7fdcfdc49700 1 -- 172.21.15.149:0/1929357656 >> 172.21.15.149:0/1929357656 conn(0x7fdcf80f09a0 msgr2=0x7fdcf805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:42.830 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.828+0000 7fdcfdc49700 1 -- 172.21.15.149:0/1929357656 shutdown_connections 2024-02-12T16:32:42.830 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:42.828+0000 7fdcfdc49700 1 -- 172.21.15.149:0/1929357656 wait complete. 2024-02-12T16:32:42.830 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:43.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:43 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:42.826887+0000 mon.smithi104 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.149:0/1929357656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:44.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:44.477 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:44.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:42.302610+0000 mgr.smithi104.ohwbhf (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:46.268 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.257+0000 7fec54a06700 1 -- 172.21.15.149:0/4074584021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec500f5410 msgr2=0x7fec500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:46.269 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.257+0000 7fec54a06700 1 --2- 172.21.15.149:0/4074584021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec500f5410 0x7fec500f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fec44009fb0 tx=0x7fec44009b30).stop 2024-02-12T16:32:46.269 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.257+0000 7fec54a06700 1 -- 172.21.15.149:0/4074584021 shutdown_connections 2024-02-12T16:32:46.269 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.257+0000 7fec54a06700 1 --2- 172.21.15.149:0/4074584021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec500f5410 0x7fec500f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:46.269 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.257+0000 7fec54a06700 1 -- 172.21.15.149:0/4074584021 >> 172.21.15.149:0/4074584021 conn(0x7fec500f0b70 msgr2=0x7fec500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:46.270 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.257+0000 7fec54a06700 1 -- 172.21.15.149:0/4074584021 shutdown_connections 2024-02-12T16:32:46.270 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.257+0000 7fec54a06700 1 -- 172.21.15.149:0/4074584021 wait complete. 2024-02-12T16:32:46.270 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.257+0000 7fec54a06700 1 Processor -- start 2024-02-12T16:32:46.270 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.258+0000 7fec54a06700 1 -- start start 2024-02-12T16:32:46.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.258+0000 7fec54a06700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec500f5410 0x7fec50109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:46.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.258+0000 7fec54a06700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec50060240 con 0x7fec500f5410 2024-02-12T16:32:46.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.258+0000 7fec4e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec500f5410 0x7fec50109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:46.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.258+0000 7fec4e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec500f5410 0x7fec50109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:47518/0 (socket says 172.21.15.149:47518) 2024-02-12T16:32:46.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.258+0000 7fec4e59c700 1 -- 172.21.15.149:0/1031714575 learned_addr learned my addr 172.21.15.149:0/1031714575 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:46.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.258+0000 7fec4e59c700 1 -- 172.21.15.149:0/1031714575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec44005040 con 0x7fec500f5410 2024-02-12T16:32:46.272 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.258+0000 7fec4e59c700 1 --2- 172.21.15.149:0/1031714575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec500f5410 0x7fec50109520 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fec44005010 tx=0x7fec4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:46.272 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.259+0000 7fec3effd700 1 -- 172.21.15.149:0/1031714575 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec44009320 con 0x7fec500f5410 2024-02-12T16:32:46.272 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.259+0000 7fec54a06700 1 -- 172.21.15.149:0/1031714575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec50060440 con 0x7fec500f5410 2024-02-12T16:32:46.272 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.259+0000 7fec3effd700 1 -- 172.21.15.149:0/1031714575 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec44013380 con 0x7fec500f5410 2024-02-12T16:32:46.272 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.259+0000 7fec54a06700 1 -- 172.21.15.149:0/1031714575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec50109bd0 con 0x7fec500f5410 2024-02-12T16:32:46.272 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.259+0000 7fec3effd700 1 -- 172.21.15.149:0/1031714575 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec44014df0 con 0x7fec500f5410 2024-02-12T16:32:46.273 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.259+0000 7fec3effd700 1 -- 172.21.15.149:0/1031714575 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec440263a0 con 0x7fec500f5410 2024-02-12T16:32:46.273 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.260+0000 7fec3effd700 1 --2- 172.21.15.149:0/1031714575 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fec3802ff10 0x7fec380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:46.273 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.260+0000 7fec3cff9700 1 -- 172.21.15.149:0/1031714575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec28004fa0 con 0x7fec500f5410 2024-02-12T16:32:46.273 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.260+0000 7fec3effd700 1 -- 172.21.15.149:0/1031714575 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec44041480 con 0x7fec500f5410 2024-02-12T16:32:46.273 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.260+0000 7fec4dd9b700 1 --2- 172.21.15.149:0/1031714575 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fec3802ff10 0x7fec380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:46.274 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.260+0000 7fec4dd9b700 1 --2- 172.21.15.149:0/1031714575 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fec3802ff10 0x7fec380323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fec400096a0 tx=0x7fec40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:46.274 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.264+0000 7fec3effd700 1 -- 172.21.15.149:0/1031714575 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec44014420 con 0x7fec500f5410 2024-02-12T16:32:46.470 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.468+0000 7fec3cff9700 1 -- 172.21.15.149:0/1031714575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec28005670 con 0x7fec500f5410 2024-02-12T16:32:46.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.468+0000 7fec3effd700 1 -- 172.21.15.149:0/1031714575 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fec44012330 con 0x7fec500f5410 2024-02-12T16:32:46.473 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:46.473 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:46.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.470+0000 7fec3cff9700 1 -- 172.21.15.149:0/1031714575 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fec3802ff10 msgr2=0x7fec380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:46.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.470+0000 7fec3cff9700 1 --2- 172.21.15.149:0/1031714575 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fec3802ff10 0x7fec380323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fec400096a0 tx=0x7fec40006b40).stop 2024-02-12T16:32:46.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.470+0000 7fec3cff9700 1 -- 172.21.15.149:0/1031714575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec500f5410 msgr2=0x7fec50109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:46.475 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.470+0000 7fec3cff9700 1 --2- 172.21.15.149:0/1031714575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec500f5410 0x7fec50109520 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fec44005010 tx=0x7fec4400b040).stop 2024-02-12T16:32:46.475 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.470+0000 7fec3cff9700 1 -- 172.21.15.149:0/1031714575 shutdown_connections 2024-02-12T16:32:46.475 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.470+0000 7fec3cff9700 1 --2- 172.21.15.149:0/1031714575 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fec3802ff10 0x7fec380323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:46.475 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.470+0000 7fec3cff9700 1 --2- 172.21.15.149:0/1031714575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec500f5410 0x7fec50109520 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:46.475 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.470+0000 7fec3cff9700 1 -- 172.21.15.149:0/1031714575 >> 172.21.15.149:0/1031714575 conn(0x7fec500f0b70 msgr2=0x7fec500f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:46.475 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.470+0000 7fec3cff9700 1 -- 172.21.15.149:0/1031714575 shutdown_connections 2024-02-12T16:32:46.475 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:46.470+0000 7fec3cff9700 1 -- 172.21.15.149:0/1031714575 wait complete. 2024-02-12T16:32:46.475 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:46.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:44.302958+0000 mgr.smithi104.ohwbhf (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:47.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:47 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:46.469755+0000 mon.smithi104 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.149:0/1031714575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:48.036 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:48.036 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:48.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:46.303334+0000 mgr.smithi104.ohwbhf (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:49.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.786+0000 7f5eadacc700 1 -- 172.21.15.149:0/3215210673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ea8060140 msgr2=0x7f5ea80f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:49.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.786+0000 7f5eadacc700 1 --2- 172.21.15.149:0/3215210673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ea8060140 0x7f5ea80f8fc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f5e9c009fb0 tx=0x7f5e9c009b30).stop 2024-02-12T16:32:49.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.786+0000 7f5eadacc700 1 -- 172.21.15.149:0/3215210673 shutdown_connections 2024-02-12T16:32:49.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.786+0000 7f5eadacc700 1 --2- 172.21.15.149:0/3215210673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ea8060140 0x7f5ea80f8fc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:49.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.786+0000 7f5eadacc700 1 -- 172.21.15.149:0/3215210673 >> 172.21.15.149:0/3215210673 conn(0x7f5ea80f09a0 msgr2=0x7f5ea80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:49.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.786+0000 7f5eadacc700 1 -- 172.21.15.149:0/3215210673 shutdown_connections 2024-02-12T16:32:49.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.786+0000 7f5eadacc700 1 -- 172.21.15.149:0/3215210673 wait complete. 2024-02-12T16:32:49.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.786+0000 7f5eadacc700 1 Processor -- start 2024-02-12T16:32:49.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.786+0000 7f5eadacc700 1 -- start start 2024-02-12T16:32:49.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.787+0000 7f5eadacc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ea8060140 0x7f5ea8106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:49.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.787+0000 7f5eadacc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea8106af0 con 0x7f5ea8060140 2024-02-12T16:32:49.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.787+0000 7f5eacaca700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ea8060140 0x7f5ea8106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:49.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.787+0000 7f5eacaca700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ea8060140 0x7f5ea8106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:47538/0 (socket says 172.21.15.149:47538) 2024-02-12T16:32:49.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.787+0000 7f5eacaca700 1 -- 172.21.15.149:0/350028644 learned_addr learned my addr 172.21.15.149:0/350028644 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.787+0000 7f5eacaca700 1 -- 172.21.15.149:0/350028644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c005040 con 0x7f5ea8060140 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.788+0000 7f5eacaca700 1 --2- 172.21.15.149:0/350028644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ea8060140 0x7f5ea8106600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f5e9c005010 tx=0x7f5e9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.788+0000 7f5ea57fa700 1 -- 172.21.15.149:0/350028644 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e9c009320 con 0x7f5ea8060140 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.788+0000 7f5eadacc700 1 -- 172.21.15.149:0/350028644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea8106cf0 con 0x7f5ea8060140 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.788+0000 7f5ea57fa700 1 -- 172.21.15.149:0/350028644 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e9c013380 con 0x7f5ea8060140 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.788+0000 7f5ea57fa700 1 -- 172.21.15.149:0/350028644 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e9c014e50 con 0x7f5ea8060140 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.788+0000 7f5eadacc700 1 -- 172.21.15.149:0/350028644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea8107180 con 0x7f5ea8060140 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.788+0000 7f5ea57fa700 1 -- 172.21.15.149:0/350028644 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e9c014420 con 0x7f5ea8060140 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.789+0000 7f5ea57fa700 1 --2- 172.21.15.149:0/350028644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5e9802ff10 0x7f5e980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.789+0000 7f5ea57fa700 1 -- 172.21.15.149:0/350028644 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e9c040f40 con 0x7f5ea8060140 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.789+0000 7f5eadacc700 1 -- 172.21.15.149:0/350028644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea8045ad0 con 0x7f5ea8060140 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.789+0000 7f5ea7fff700 1 --2- 172.21.15.149:0/350028644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5e9802ff10 0x7f5e980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:49.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.789+0000 7f5ea7fff700 1 --2- 172.21.15.149:0/350028644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5e9802ff10 0x7f5e980323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f5e940096a0 tx=0x7f5e94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:49.795 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.794+0000 7f5ea57fa700 1 -- 172.21.15.149:0/350028644 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e9c0095f0 con 0x7f5ea8060140 2024-02-12T16:32:50.000 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.998+0000 7f5eadacc700 1 -- 172.21.15.149:0/350028644 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ea8107ea0 con 0x7f5ea8060140 2024-02-12T16:32:50.000 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:49.998+0000 7f5ea57fa700 1 -- 172.21.15.149:0/350028644 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5e9c02b020 con 0x7f5ea8060140 2024-02-12T16:32:50.001 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:50.001 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:50.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:50.000+0000 7f5eadacc700 1 -- 172.21.15.149:0/350028644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5e9802ff10 msgr2=0x7f5e980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:50.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:50.000+0000 7f5eadacc700 1 --2- 172.21.15.149:0/350028644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5e9802ff10 0x7f5e980323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f5e940096a0 tx=0x7f5e94006b40).stop 2024-02-12T16:32:50.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:50.000+0000 7f5eadacc700 1 -- 172.21.15.149:0/350028644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ea8060140 msgr2=0x7f5ea8106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:50.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:50.000+0000 7f5eadacc700 1 --2- 172.21.15.149:0/350028644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ea8060140 0x7f5ea8106600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f5e9c005010 tx=0x7f5e9c00b040).stop 2024-02-12T16:32:50.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:50.000+0000 7f5eadacc700 1 -- 172.21.15.149:0/350028644 shutdown_connections 2024-02-12T16:32:50.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:50.000+0000 7f5eadacc700 1 --2- 172.21.15.149:0/350028644 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5e9802ff10 0x7f5e980323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:50.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:50.000+0000 7f5eadacc700 1 --2- 172.21.15.149:0/350028644 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ea8060140 0x7f5ea8106600 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:50.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:50.000+0000 7f5eadacc700 1 -- 172.21.15.149:0/350028644 >> 172.21.15.149:0/350028644 conn(0x7f5ea80f09a0 msgr2=0x7f5ea80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:50.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:50.000+0000 7f5eadacc700 1 -- 172.21.15.149:0/350028644 shutdown_connections 2024-02-12T16:32:50.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:50.001+0000 7f5eadacc700 1 -- 172.21.15.149:0/350028644 wait complete. 2024-02-12T16:32:50.004 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:50.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:48.303717+0000 mgr.smithi104.ohwbhf (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:50.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:49.999845+0000 mon.smithi104 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.149:0/350028644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:51.638 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:51.638 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:52.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:50.304033+0000 mgr.smithi104.ohwbhf (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:53.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.363+0000 7f726fcda700 1 -- 172.21.15.149:0/3232312861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72680f5410 msgr2=0x7f72680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:53.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.363+0000 7f726fcda700 1 --2- 172.21.15.149:0/3232312861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72680f5410 0x7f72680f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f7258009fb0 tx=0x7f7258009b30).stop 2024-02-12T16:32:53.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.363+0000 7f726fcda700 1 -- 172.21.15.149:0/3232312861 shutdown_connections 2024-02-12T16:32:53.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.363+0000 7f726fcda700 1 --2- 172.21.15.149:0/3232312861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72680f5410 0x7f72680f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:53.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.363+0000 7f726fcda700 1 -- 172.21.15.149:0/3232312861 >> 172.21.15.149:0/3232312861 conn(0x7f72680f0b70 msgr2=0x7f72680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:53.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.363+0000 7f726fcda700 1 -- 172.21.15.149:0/3232312861 shutdown_connections 2024-02-12T16:32:53.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.363+0000 7f726fcda700 1 -- 172.21.15.149:0/3232312861 wait complete. 2024-02-12T16:32:53.372 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.364+0000 7f726fcda700 1 Processor -- start 2024-02-12T16:32:53.372 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.364+0000 7f726fcda700 1 -- start start 2024-02-12T16:32:53.372 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.364+0000 7f726fcda700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72680f5410 0x7f7268109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.364+0000 7f726fcda700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7268060220 con 0x7f72680f5410 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.364+0000 7f726da76700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72680f5410 0x7f7268109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.364+0000 7f726da76700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72680f5410 0x7f7268109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:47544/0 (socket says 172.21.15.149:47544) 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.364+0000 7f726da76700 1 -- 172.21.15.149:0/3610713499 learned_addr learned my addr 172.21.15.149:0/3610713499 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.364+0000 7f726da76700 1 -- 172.21.15.149:0/3610713499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7258005040 con 0x7f72680f5410 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.365+0000 7f726da76700 1 --2- 172.21.15.149:0/3610713499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72680f5410 0x7f7268109500 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f7258005010 tx=0x7f725800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.365+0000 7f725e7fc700 1 -- 172.21.15.149:0/3610713499 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7258009320 con 0x7f72680f5410 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.365+0000 7f725e7fc700 1 -- 172.21.15.149:0/3610713499 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7258013380 con 0x7f72680f5410 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.365+0000 7f726fcda700 1 -- 172.21.15.149:0/3610713499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7268060420 con 0x7f72680f5410 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.365+0000 7f725e7fc700 1 -- 172.21.15.149:0/3610713499 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7258014e50 con 0x7f72680f5410 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.365+0000 7f726fcda700 1 -- 172.21.15.149:0/3610713499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7268109bb0 con 0x7f72680f5410 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.366+0000 7f725e7fc700 1 -- 172.21.15.149:0/3610713499 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7258014420 con 0x7f72680f5410 2024-02-12T16:32:53.373 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.366+0000 7f726fcda700 1 -- 172.21.15.149:0/3610713499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7268059100 con 0x7f72680f5410 2024-02-12T16:32:53.374 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.366+0000 7f725e7fc700 1 --2- 172.21.15.149:0/3610713499 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f725402ff10 0x7f72540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:53.374 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.366+0000 7f725e7fc700 1 -- 172.21.15.149:0/3610713499 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7258040e30 con 0x7f72680f5410 2024-02-12T16:32:53.374 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.366+0000 7f726d275700 1 --2- 172.21.15.149:0/3610713499 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f725402ff10 0x7f72540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:53.374 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.366+0000 7f726d275700 1 --2- 172.21.15.149:0/3610713499 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f725402ff10 0x7f72540323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f72640096a0 tx=0x7f7264006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:53.374 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.370+0000 7f725e7fc700 1 -- 172.21.15.149:0/3610713499 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72580095f0 con 0x7f72680f5410 2024-02-12T16:32:53.576 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.574+0000 7f726fcda700 1 -- 172.21.15.149:0/3610713499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7268045ad0 con 0x7f72680f5410 2024-02-12T16:32:53.577 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.575+0000 7f725e7fc700 1 -- 172.21.15.149:0/3610713499 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f725801db70 con 0x7f72680f5410 2024-02-12T16:32:53.577 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:53.577 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:53.579 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.577+0000 7f726fcda700 1 -- 172.21.15.149:0/3610713499 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f725402ff10 msgr2=0x7f72540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:53.579 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.577+0000 7f726fcda700 1 --2- 172.21.15.149:0/3610713499 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f725402ff10 0x7f72540323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f72640096a0 tx=0x7f7264006b40).stop 2024-02-12T16:32:53.579 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.577+0000 7f726fcda700 1 -- 172.21.15.149:0/3610713499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72680f5410 msgr2=0x7f7268109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:53.579 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.577+0000 7f726fcda700 1 --2- 172.21.15.149:0/3610713499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72680f5410 0x7f7268109500 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f7258005010 tx=0x7f725800b040).stop 2024-02-12T16:32:53.579 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.577+0000 7f726fcda700 1 -- 172.21.15.149:0/3610713499 shutdown_connections 2024-02-12T16:32:53.579 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.577+0000 7f726fcda700 1 --2- 172.21.15.149:0/3610713499 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f725402ff10 0x7f72540323d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:53.579 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.577+0000 7f726fcda700 1 --2- 172.21.15.149:0/3610713499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f72680f5410 0x7f7268109500 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:53.579 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.577+0000 7f726fcda700 1 -- 172.21.15.149:0/3610713499 >> 172.21.15.149:0/3610713499 conn(0x7f72680f0b70 msgr2=0x7f72680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:53.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.577+0000 7f726fcda700 1 -- 172.21.15.149:0/3610713499 shutdown_connections 2024-02-12T16:32:53.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:53.577+0000 7f726fcda700 1 -- 172.21.15.149:0/3610713499 wait complete. 2024-02-12T16:32:53.580 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:54.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:52.304243+0000 mgr.smithi104.ohwbhf (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:54.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:53.576528+0000 mon.smithi104 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.149:0/3610713499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:55.381 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:55.382 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:32:56.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:54.304476+0000 mgr.smithi104.ohwbhf (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:57.223 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.220+0000 7fc2d8684700 1 -- 172.21.15.149:0/3562862508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00f6510 msgr2=0x7fc2d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:57.223 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.220+0000 7fc2d8684700 1 --2- 172.21.15.149:0/3562862508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00f6510 0x7fc2d00f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fc2c0009fb0 tx=0x7fc2c0009b30).stop 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.220+0000 7fc2d8684700 1 -- 172.21.15.149:0/3562862508 shutdown_connections 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.220+0000 7fc2d8684700 1 --2- 172.21.15.149:0/3562862508 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00f6510 0x7fc2d00f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.220+0000 7fc2d8684700 1 -- 172.21.15.149:0/3562862508 >> 172.21.15.149:0/3562862508 conn(0x7fc2d00f1c70 msgr2=0x7fc2d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.221+0000 7fc2d8684700 1 -- 172.21.15.149:0/3562862508 shutdown_connections 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.221+0000 7fc2d8684700 1 -- 172.21.15.149:0/3562862508 wait complete. 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.221+0000 7fc2d8684700 1 Processor -- start 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.221+0000 7fc2d8684700 1 -- start start 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.221+0000 7fc2d8684700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00f6510 0x7fc2d0108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.221+0000 7fc2d8684700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d0108ef0 con 0x7fc2d00f6510 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.221+0000 7fc2d6420700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00f6510 0x7fc2d0108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.221+0000 7fc2d6420700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00f6510 0x7fc2d0108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33458/0 (socket says 172.21.15.149:33458) 2024-02-12T16:32:57.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.221+0000 7fc2d6420700 1 -- 172.21.15.149:0/2061292936 learned_addr learned my addr 172.21.15.149:0/2061292936 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:32:57.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.222+0000 7fc2d6420700 1 -- 172.21.15.149:0/2061292936 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c0005040 con 0x7fc2d00f6510 2024-02-12T16:32:57.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.222+0000 7fc2d6420700 1 --2- 172.21.15.149:0/2061292936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00f6510 0x7fc2d0108a00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc2c0005010 tx=0x7fc2c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:57.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.222+0000 7fc2c6ffd700 1 -- 172.21.15.149:0/2061292936 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c0009320 con 0x7fc2d00f6510 2024-02-12T16:32:57.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.222+0000 7fc2d8684700 1 -- 172.21.15.149:0/2061292936 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d01090f0 con 0x7fc2d00f6510 2024-02-12T16:32:57.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.222+0000 7fc2c6ffd700 1 -- 172.21.15.149:0/2061292936 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2c0013380 con 0x7fc2d00f6510 2024-02-12T16:32:57.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.222+0000 7fc2c6ffd700 1 -- 172.21.15.149:0/2061292936 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c0014e50 con 0x7fc2d00f6510 2024-02-12T16:32:57.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.222+0000 7fc2d8684700 1 -- 172.21.15.149:0/2061292936 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d0109580 con 0x7fc2d00f6510 2024-02-12T16:32:57.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.223+0000 7fc2c6ffd700 1 -- 172.21.15.149:0/2061292936 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2c00095f0 con 0x7fc2d00f6510 2024-02-12T16:32:57.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.223+0000 7fc2c6ffd700 1 --2- 172.21.15.149:0/2061292936 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc2bc02ff10 0x7fc2bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:32:57.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.223+0000 7fc2d8684700 1 -- 172.21.15.149:0/2061292936 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d0045ad0 con 0x7fc2d00f6510 2024-02-12T16:32:57.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.223+0000 7fc2c6ffd700 1 -- 172.21.15.149:0/2061292936 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2c0040e70 con 0x7fc2d00f6510 2024-02-12T16:32:57.227 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.223+0000 7fc2d5c1f700 1 --2- 172.21.15.149:0/2061292936 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc2bc02ff10 0x7fc2bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:32:57.227 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.224+0000 7fc2d5c1f700 1 --2- 172.21.15.149:0/2061292936 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc2bc02ff10 0x7fc2bc0323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fc2cc0096a0 tx=0x7fc2cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:32:57.230 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.228+0000 7fc2c6ffd700 1 -- 172.21.15.149:0/2061292936 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2c001d990 con 0x7fc2d00f6510 2024-02-12T16:32:57.435 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.433+0000 7fc2d8684700 1 -- 172.21.15.149:0/2061292936 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2d00fc190 con 0x7fc2d00f6510 2024-02-12T16:32:57.435 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.433+0000 7fc2c6ffd700 1 -- 172.21.15.149:0/2061292936 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc2c0010230 con 0x7fc2d00f6510 2024-02-12T16:32:57.436 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:32:57.436 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:32:57.437 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.435+0000 7fc2d8684700 1 -- 172.21.15.149:0/2061292936 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc2bc02ff10 msgr2=0x7fc2bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:57.437 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.435+0000 7fc2d8684700 1 --2- 172.21.15.149:0/2061292936 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc2bc02ff10 0x7fc2bc0323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fc2cc0096a0 tx=0x7fc2cc006b40).stop 2024-02-12T16:32:57.437 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.435+0000 7fc2d8684700 1 -- 172.21.15.149:0/2061292936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00f6510 msgr2=0x7fc2d0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:32:57.437 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.435+0000 7fc2d8684700 1 --2- 172.21.15.149:0/2061292936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00f6510 0x7fc2d0108a00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc2c0005010 tx=0x7fc2c000b040).stop 2024-02-12T16:32:57.437 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.435+0000 7fc2d8684700 1 -- 172.21.15.149:0/2061292936 shutdown_connections 2024-02-12T16:32:57.437 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.435+0000 7fc2d8684700 1 --2- 172.21.15.149:0/2061292936 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc2bc02ff10 0x7fc2bc0323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:57.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.435+0000 7fc2d8684700 1 --2- 172.21.15.149:0/2061292936 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc2d00f6510 0x7fc2d0108a00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:32:57.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.435+0000 7fc2d8684700 1 -- 172.21.15.149:0/2061292936 >> 172.21.15.149:0/2061292936 conn(0x7fc2d00f1c70 msgr2=0x7fc2d00f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:32:57.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.435+0000 7fc2d8684700 1 -- 172.21.15.149:0/2061292936 shutdown_connections 2024-02-12T16:32:57.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:32:57.435+0000 7fc2d8684700 1 -- 172.21.15.149:0/2061292936 wait complete. 2024-02-12T16:32:57.438 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:32:58.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:56.304722+0000 mgr.smithi104.ohwbhf (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:32:58.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:32:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:32:57.434512+0000 mon.smithi104 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.149:0/2061292936' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:32:59.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:32:59.062 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:00.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:32:58.305015+0000 mgr.smithi104.ohwbhf (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:00.871 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.867+0000 7fe2226de700 1 -- 172.21.15.149:0/2687238659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe21c0f5410 msgr2=0x7fe21c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:00.871 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.867+0000 7fe2226de700 1 --2- 172.21.15.149:0/2687238659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe21c0f5410 0x7fe21c0f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fe20c009fb0 tx=0x7fe20c009b30).stop 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.867+0000 7fe2226de700 1 -- 172.21.15.149:0/2687238659 shutdown_connections 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.867+0000 7fe2226de700 1 --2- 172.21.15.149:0/2687238659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe21c0f5410 0x7fe21c0f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.867+0000 7fe2226de700 1 -- 172.21.15.149:0/2687238659 >> 172.21.15.149:0/2687238659 conn(0x7fe21c0f0b70 msgr2=0x7fe21c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.867+0000 7fe2226de700 1 -- 172.21.15.149:0/2687238659 shutdown_connections 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.867+0000 7fe2226de700 1 -- 172.21.15.149:0/2687238659 wait complete. 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.868+0000 7fe2226de700 1 Processor -- start 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.868+0000 7fe2226de700 1 -- start start 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe2226de700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe21c0f5410 0x7fe21c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe2226de700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c106cc0 con 0x7fe21c0f5410 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe21bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe21c0f5410 0x7fe21c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe21bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe21c0f5410 0x7fe21c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33472/0 (socket says 172.21.15.149:33472) 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe21bfff700 1 -- 172.21.15.149:0/413585603 learned_addr learned my addr 172.21.15.149:0/413585603 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:00.872 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe21bfff700 1 -- 172.21.15.149:0/413585603 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c005040 con 0x7fe21c0f5410 2024-02-12T16:33:00.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe21bfff700 1 --2- 172.21.15.149:0/413585603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe21c0f5410 0x7fe21c1067d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe20c005010 tx=0x7fe20c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:00.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe218ff9700 1 -- 172.21.15.149:0/413585603 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe20c009320 con 0x7fe21c0f5410 2024-02-12T16:33:00.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe2226de700 1 -- 172.21.15.149:0/413585603 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c106ec0 con 0x7fe21c0f5410 2024-02-12T16:33:00.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe218ff9700 1 -- 172.21.15.149:0/413585603 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe20c013380 con 0x7fe21c0f5410 2024-02-12T16:33:00.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe218ff9700 1 -- 172.21.15.149:0/413585603 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe20c014e50 con 0x7fe21c0f5410 2024-02-12T16:33:00.873 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.869+0000 7fe2226de700 1 -- 172.21.15.149:0/413585603 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c107350 con 0x7fe21c0f5410 2024-02-12T16:33:00.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.870+0000 7fe218ff9700 1 -- 172.21.15.149:0/413585603 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe20c0095f0 con 0x7fe21c0f5410 2024-02-12T16:33:00.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.870+0000 7fe2226de700 1 -- 172.21.15.149:0/413585603 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c045ad0 con 0x7fe21c0f5410 2024-02-12T16:33:00.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.870+0000 7fe218ff9700 1 --2- 172.21.15.149:0/413585603 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe20402ff10 0x7fe2040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:00.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.870+0000 7fe218ff9700 1 -- 172.21.15.149:0/413585603 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe20c040e90 con 0x7fe21c0f5410 2024-02-12T16:33:00.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.870+0000 7fe21b7fe700 1 --2- 172.21.15.149:0/413585603 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe20402ff10 0x7fe2040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:00.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.871+0000 7fe21b7fe700 1 --2- 172.21.15.149:0/413585603 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe20402ff10 0x7fe2040323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe2100096a0 tx=0x7fe210006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:00.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:00.875+0000 7fe218ff9700 1 -- 172.21.15.149:0/413585603 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe20c014420 con 0x7fe21c0f5410 2024-02-12T16:33:01.078 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.077+0000 7fe2226de700 1 -- 172.21.15.149:0/413585603 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe21c108200 con 0x7fe21c0f5410 2024-02-12T16:33:01.080 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.077+0000 7fe218ff9700 1 -- 172.21.15.149:0/413585603 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe20c010230 con 0x7fe21c0f5410 2024-02-12T16:33:01.080 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:01.080 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:01.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.079+0000 7fe2226de700 1 -- 172.21.15.149:0/413585603 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe20402ff10 msgr2=0x7fe2040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:01.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.079+0000 7fe2226de700 1 --2- 172.21.15.149:0/413585603 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe20402ff10 0x7fe2040323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe2100096a0 tx=0x7fe210006b40).stop 2024-02-12T16:33:01.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.079+0000 7fe2226de700 1 -- 172.21.15.149:0/413585603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe21c0f5410 msgr2=0x7fe21c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:01.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.079+0000 7fe2226de700 1 --2- 172.21.15.149:0/413585603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe21c0f5410 0x7fe21c1067d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe20c005010 tx=0x7fe20c00b040).stop 2024-02-12T16:33:01.082 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.079+0000 7fe2226de700 1 -- 172.21.15.149:0/413585603 shutdown_connections 2024-02-12T16:33:01.082 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.079+0000 7fe2226de700 1 --2- 172.21.15.149:0/413585603 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe20402ff10 0x7fe2040323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:01.082 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.079+0000 7fe2226de700 1 --2- 172.21.15.149:0/413585603 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe21c0f5410 0x7fe21c1067d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:01.082 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.079+0000 7fe2226de700 1 -- 172.21.15.149:0/413585603 >> 172.21.15.149:0/413585603 conn(0x7fe21c0f0b70 msgr2=0x7fe21c0f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:01.082 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.079+0000 7fe2226de700 1 -- 172.21.15.149:0/413585603 shutdown_connections 2024-02-12T16:33:01.082 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:01.079+0000 7fe2226de700 1 -- 172.21.15.149:0/413585603 wait complete. 2024-02-12T16:33:01.082 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:01.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:01.078578+0000 mon.smithi104 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.149:0/413585603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:02.614 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:02.615 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:02.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:00.305271+0000 mgr.smithi104.ohwbhf (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:04.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.399+0000 7fde10bc5700 1 -- 172.21.15.149:0/1875080526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0c0601c0 msgr2=0x7fde0c0f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:04.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.399+0000 7fde10bc5700 1 --2- 172.21.15.149:0/1875080526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0c0601c0 0x7fde0c0f91b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fddf4009fb0 tx=0x7fddf4009b30).stop 2024-02-12T16:33:04.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.400+0000 7fde10bc5700 1 -- 172.21.15.149:0/1875080526 shutdown_connections 2024-02-12T16:33:04.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.400+0000 7fde10bc5700 1 --2- 172.21.15.149:0/1875080526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0c0601c0 0x7fde0c0f91b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:04.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.400+0000 7fde10bc5700 1 -- 172.21.15.149:0/1875080526 >> 172.21.15.149:0/1875080526 conn(0x7fde0c0f0b70 msgr2=0x7fde0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:04.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.400+0000 7fde10bc5700 1 -- 172.21.15.149:0/1875080526 shutdown_connections 2024-02-12T16:33:04.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.400+0000 7fde10bc5700 1 -- 172.21.15.149:0/1875080526 wait complete. 2024-02-12T16:33:04.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.400+0000 7fde10bc5700 1 Processor -- start 2024-02-12T16:33:04.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.400+0000 7fde10bc5700 1 -- start start 2024-02-12T16:33:04.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde10bc5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0c0601c0 0x7fde0c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:04.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde10bc5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c106cb0 con 0x7fde0c0601c0 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde0a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0c0601c0 0x7fde0c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde0a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0c0601c0 0x7fde0c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:37268/0 (socket says 172.21.15.149:37268) 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde0a59c700 1 -- 172.21.15.149:0/3079081722 learned_addr learned my addr 172.21.15.149:0/3079081722 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde0a59c700 1 -- 172.21.15.149:0/3079081722 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf4005040 con 0x7fde0c0601c0 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde0a59c700 1 --2- 172.21.15.149:0/3079081722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0c0601c0 0x7fde0c1067c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fddf4009830 tx=0x7fddf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde02ffd700 1 -- 172.21.15.149:0/3079081722 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddf4009320 con 0x7fde0c0601c0 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde10bc5700 1 -- 172.21.15.149:0/3079081722 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c106eb0 con 0x7fde0c0601c0 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde02ffd700 1 -- 172.21.15.149:0/3079081722 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddf4013380 con 0x7fde0c0601c0 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde02ffd700 1 -- 172.21.15.149:0/3079081722 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddf4014420 con 0x7fde0c0601c0 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.401+0000 7fde10bc5700 1 -- 172.21.15.149:0/3079081722 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c107340 con 0x7fde0c0601c0 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.402+0000 7fde02ffd700 1 -- 172.21.15.149:0/3079081722 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fddf401e420 con 0x7fde0c0601c0 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.402+0000 7fde10bc5700 1 -- 172.21.15.149:0/3079081722 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0c045ad0 con 0x7fde0c0601c0 2024-02-12T16:33:04.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.402+0000 7fde02ffd700 1 --2- 172.21.15.149:0/3079081722 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fddf802ff60 0x7fddf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:04.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.402+0000 7fde02ffd700 1 -- 172.21.15.149:0/3079081722 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fddf4040eb0 con 0x7fde0c0601c0 2024-02-12T16:33:04.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.403+0000 7fde09d9b700 1 --2- 172.21.15.149:0/3079081722 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fddf802ff60 0x7fddf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:04.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.403+0000 7fde09d9b700 1 --2- 172.21.15.149:0/3079081722 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fddf802ff60 0x7fddf8032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fddfc0096a0 tx=0x7fddfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:04.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.407+0000 7fde02ffd700 1 -- 172.21.15.149:0/3079081722 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fddf401cb10 con 0x7fde0c0601c0 2024-02-12T16:33:04.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.612+0000 7fde10bc5700 1 -- 172.21.15.149:0/3079081722 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde0c0fc090 con 0x7fde0c0601c0 2024-02-12T16:33:04.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.613+0000 7fde02ffd700 1 -- 172.21.15.149:0/3079081722 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fddf4004490 con 0x7fde0c0601c0 2024-02-12T16:33:04.615 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:04.615 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:04.616 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.614+0000 7fde10bc5700 1 -- 172.21.15.149:0/3079081722 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fddf802ff60 msgr2=0x7fddf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:04.616 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.614+0000 7fde10bc5700 1 --2- 172.21.15.149:0/3079081722 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fddf802ff60 0x7fddf8032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fddfc0096a0 tx=0x7fddfc006b40).stop 2024-02-12T16:33:04.616 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.614+0000 7fde10bc5700 1 -- 172.21.15.149:0/3079081722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0c0601c0 msgr2=0x7fde0c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:04.616 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.614+0000 7fde10bc5700 1 --2- 172.21.15.149:0/3079081722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0c0601c0 0x7fde0c1067c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fddf4009830 tx=0x7fddf400b040).stop 2024-02-12T16:33:04.616 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.614+0000 7fde10bc5700 1 -- 172.21.15.149:0/3079081722 shutdown_connections 2024-02-12T16:33:04.617 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.614+0000 7fde10bc5700 1 --2- 172.21.15.149:0/3079081722 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fddf802ff60 0x7fddf8032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:04.617 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.614+0000 7fde10bc5700 1 --2- 172.21.15.149:0/3079081722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde0c0601c0 0x7fde0c1067c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:04.617 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.614+0000 7fde10bc5700 1 -- 172.21.15.149:0/3079081722 >> 172.21.15.149:0/3079081722 conn(0x7fde0c0f0b70 msgr2=0x7fde0c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:04.617 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.614+0000 7fde10bc5700 1 -- 172.21.15.149:0/3079081722 shutdown_connections 2024-02-12T16:33:04.617 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:04.614+0000 7fde10bc5700 1 -- 172.21.15.149:0/3079081722 wait complete. 2024-02-12T16:33:04.617 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:04.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:02.305583+0000 mgr.smithi104.ohwbhf (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:05.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:04.613874+0000 mon.smithi104 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.149:0/3079081722' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:06.203 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:06.203 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:06.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:04.305906+0000 mgr.smithi104.ohwbhf (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:07.945 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.942+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1168276480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ae00f5410 msgr2=0x7f5ae00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.942+0000 7f5ae59ae700 1 --2- 172.21.15.149:0/1168276480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ae00f5410 0x7f5ae00f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f5ad0009fb0 tx=0x7f5ad0009b30).stop 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.942+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1168276480 shutdown_connections 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.942+0000 7f5ae59ae700 1 --2- 172.21.15.149:0/1168276480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ae00f5410 0x7f5ae00f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.942+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1168276480 >> 172.21.15.149:0/1168276480 conn(0x7f5ae00f0b70 msgr2=0x7f5ae00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.942+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1168276480 shutdown_connections 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.942+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1168276480 wait complete. 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.943+0000 7f5ae59ae700 1 Processor -- start 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.943+0000 7f5ae59ae700 1 -- start start 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.943+0000 7f5ae59ae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ae00f5410 0x7f5ae01094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.943+0000 7f5ae59ae700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae0060240 con 0x7f5ae00f5410 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.943+0000 7f5adeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ae00f5410 0x7f5ae01094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:07.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.943+0000 7f5adeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ae00f5410 0x7f5ae01094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:37282/0 (socket says 172.21.15.149:37282) 2024-02-12T16:33:07.947 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.943+0000 7f5adeffd700 1 -- 172.21.15.149:0/1132209378 learned_addr learned my addr 172.21.15.149:0/1132209378 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:07.947 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.944+0000 7f5adeffd700 1 -- 172.21.15.149:0/1132209378 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad0005040 con 0x7f5ae00f5410 2024-02-12T16:33:07.947 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.944+0000 7f5adeffd700 1 --2- 172.21.15.149:0/1132209378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ae00f5410 0x7f5ae01094e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f5ad0005010 tx=0x7f5ad000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:07.948 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.944+0000 7f5ac77fe700 1 -- 172.21.15.149:0/1132209378 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ad0009320 con 0x7f5ae00f5410 2024-02-12T16:33:07.948 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.944+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1132209378 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae0060440 con 0x7f5ae00f5410 2024-02-12T16:33:07.948 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.944+0000 7f5ac77fe700 1 -- 172.21.15.149:0/1132209378 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ad0013380 con 0x7f5ae00f5410 2024-02-12T16:33:07.948 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.944+0000 7f5ac77fe700 1 -- 172.21.15.149:0/1132209378 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ad0014e50 con 0x7f5ae00f5410 2024-02-12T16:33:07.948 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.944+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1132209378 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae0109b90 con 0x7f5ae00f5410 2024-02-12T16:33:07.949 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.945+0000 7f5ac77fe700 1 -- 172.21.15.149:0/1132209378 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ad00263a0 con 0x7f5ae00f5410 2024-02-12T16:33:07.949 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.945+0000 7f5ac77fe700 1 --2- 172.21.15.149:0/1132209378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5ac802ff10 0x7f5ac80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:07.949 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.945+0000 7f5ac77fe700 1 -- 172.21.15.149:0/1132209378 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ad0041250 con 0x7f5ae00f5410 2024-02-12T16:33:07.949 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.945+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1132209378 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae0045ad0 con 0x7f5ae00f5410 2024-02-12T16:33:07.949 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.945+0000 7f5ade7fc700 1 --2- 172.21.15.149:0/1132209378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5ac802ff10 0x7f5ac80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:07.949 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.946+0000 7f5ade7fc700 1 --2- 172.21.15.149:0/1132209378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5ac802ff10 0x7f5ac80323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5ad40096a0 tx=0x7f5ad4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:07.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:07.950+0000 7f5ac77fe700 1 -- 172.21.15.149:0/1132209378 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ad001cb10 con 0x7f5ae00f5410 2024-02-12T16:33:08.155 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.153+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1132209378 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ae010a1a0 con 0x7f5ae00f5410 2024-02-12T16:33:08.156 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.154+0000 7f5ac77fe700 1 -- 172.21.15.149:0/1132209378 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5ad001cb10 con 0x7f5ae00f5410 2024-02-12T16:33:08.156 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:08.156 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:08.157 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.155+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1132209378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5ac802ff10 msgr2=0x7f5ac80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:08.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.155+0000 7f5ae59ae700 1 --2- 172.21.15.149:0/1132209378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5ac802ff10 0x7f5ac80323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5ad40096a0 tx=0x7f5ad4006b40).stop 2024-02-12T16:33:08.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.155+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1132209378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ae00f5410 msgr2=0x7f5ae01094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:08.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.155+0000 7f5ae59ae700 1 --2- 172.21.15.149:0/1132209378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ae00f5410 0x7f5ae01094e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f5ad0005010 tx=0x7f5ad000b040).stop 2024-02-12T16:33:08.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.155+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1132209378 shutdown_connections 2024-02-12T16:33:08.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.156+0000 7f5ae59ae700 1 --2- 172.21.15.149:0/1132209378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5ac802ff10 0x7f5ac80323d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:08.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.156+0000 7f5ae59ae700 1 --2- 172.21.15.149:0/1132209378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5ae00f5410 0x7f5ae01094e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:08.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.156+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1132209378 >> 172.21.15.149:0/1132209378 conn(0x7f5ae00f0b70 msgr2=0x7f5ae00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:08.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.156+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1132209378 shutdown_connections 2024-02-12T16:33:08.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:08.156+0000 7f5ae59ae700 1 -- 172.21.15.149:0/1132209378 wait complete. 2024-02-12T16:33:08.159 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:08.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:06.306262+0000 mgr.smithi104.ohwbhf (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:08.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:08.155138+0000 mon.smithi104 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.149:0/1132209378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:09.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:09.821 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:10.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:08.306642+0000 mgr.smithi104.ohwbhf (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:11.539 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.535+0000 7f3872503700 1 -- 172.21.15.149:0/2545148108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f386c0f5410 msgr2=0x7f386c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:11.539 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.535+0000 7f3872503700 1 --2- 172.21.15.149:0/2545148108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f386c0f5410 0x7f386c0f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f385c009fb0 tx=0x7f385c009b30).stop 2024-02-12T16:33:11.539 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.535+0000 7f3872503700 1 -- 172.21.15.149:0/2545148108 shutdown_connections 2024-02-12T16:33:11.539 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.535+0000 7f3872503700 1 --2- 172.21.15.149:0/2545148108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f386c0f5410 0x7f386c0f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:11.539 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.535+0000 7f3872503700 1 -- 172.21.15.149:0/2545148108 >> 172.21.15.149:0/2545148108 conn(0x7f386c0f0b70 msgr2=0x7f386c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:11.539 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.535+0000 7f3872503700 1 -- 172.21.15.149:0/2545148108 shutdown_connections 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.535+0000 7f3872503700 1 -- 172.21.15.149:0/2545148108 wait complete. 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.536+0000 7f3872503700 1 Processor -- start 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.536+0000 7f3872503700 1 -- start start 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.536+0000 7f3872503700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f386c0f5410 0x7f386c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.536+0000 7f3872503700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c0fb960 con 0x7f386c0f5410 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.536+0000 7f386bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f386c0f5410 0x7f386c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.536+0000 7f386bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f386c0f5410 0x7f386c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:37304/0 (socket says 172.21.15.149:37304) 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.536+0000 7f386bfff700 1 -- 172.21.15.149:0/2818488005 learned_addr learned my addr 172.21.15.149:0/2818488005 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.537+0000 7f386bfff700 1 -- 172.21.15.149:0/2818488005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c005040 con 0x7f386c0f5410 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.537+0000 7f386bfff700 1 --2- 172.21.15.149:0/2818488005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f386c0f5410 0x7f386c0fd4e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f385c005010 tx=0x7f385c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.537+0000 7f3868ff9700 1 -- 172.21.15.149:0/2818488005 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f385c009320 con 0x7f386c0f5410 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.537+0000 7f3872503700 1 -- 172.21.15.149:0/2818488005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386c0fbb00 con 0x7f386c0f5410 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.537+0000 7f3868ff9700 1 -- 172.21.15.149:0/2818488005 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f385c013380 con 0x7f386c0f5410 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.537+0000 7f3868ff9700 1 -- 172.21.15.149:0/2818488005 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f385c014e50 con 0x7f386c0f5410 2024-02-12T16:33:11.540 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.537+0000 7f3872503700 1 -- 172.21.15.149:0/2818488005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386c0fbf90 con 0x7f386c0f5410 2024-02-12T16:33:11.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.538+0000 7f3868ff9700 1 -- 172.21.15.149:0/2818488005 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f385c0095f0 con 0x7f386c0f5410 2024-02-12T16:33:11.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.538+0000 7f3872503700 1 -- 172.21.15.149:0/2818488005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386c059100 con 0x7f386c0f5410 2024-02-12T16:33:11.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.538+0000 7f3868ff9700 1 --2- 172.21.15.149:0/2818488005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f385402ff10 0x7f38540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:11.542 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.538+0000 7f3868ff9700 1 -- 172.21.15.149:0/2818488005 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f385c040fa0 con 0x7f386c0f5410 2024-02-12T16:33:11.542 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.538+0000 7f386b7fe700 1 --2- 172.21.15.149:0/2818488005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f385402ff10 0x7f38540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:11.542 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.539+0000 7f386b7fe700 1 --2- 172.21.15.149:0/2818488005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f385402ff10 0x7f38540323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f38600096a0 tx=0x7f3860006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:11.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.543+0000 7f3868ff9700 1 -- 172.21.15.149:0/2818488005 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f385c014420 con 0x7f386c0f5410 2024-02-12T16:33:11.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.755+0000 7f3872503700 1 -- 172.21.15.149:0/2818488005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f386c0fafa0 con 0x7f386c0f5410 2024-02-12T16:33:11.758 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:11.758 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:11.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.755+0000 7f3868ff9700 1 -- 172.21.15.149:0/2818488005 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f385c010230 con 0x7f386c0f5410 2024-02-12T16:33:11.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.757+0000 7f3872503700 1 -- 172.21.15.149:0/2818488005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f385402ff10 msgr2=0x7f38540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:11.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.757+0000 7f3872503700 1 --2- 172.21.15.149:0/2818488005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f385402ff10 0x7f38540323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f38600096a0 tx=0x7f3860006b40).stop 2024-02-12T16:33:11.760 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.757+0000 7f3872503700 1 -- 172.21.15.149:0/2818488005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f386c0f5410 msgr2=0x7f386c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:11.760 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.757+0000 7f3872503700 1 --2- 172.21.15.149:0/2818488005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f386c0f5410 0x7f386c0fd4e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f385c005010 tx=0x7f385c00b040).stop 2024-02-12T16:33:11.760 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.757+0000 7f3872503700 1 -- 172.21.15.149:0/2818488005 shutdown_connections 2024-02-12T16:33:11.760 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.757+0000 7f3872503700 1 --2- 172.21.15.149:0/2818488005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f385402ff10 0x7f38540323d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:11.760 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.757+0000 7f3872503700 1 --2- 172.21.15.149:0/2818488005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f386c0f5410 0x7f386c0fd4e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:11.760 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.757+0000 7f3872503700 1 -- 172.21.15.149:0/2818488005 >> 172.21.15.149:0/2818488005 conn(0x7f386c0f0b70 msgr2=0x7f386c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:11.760 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.757+0000 7f3872503700 1 -- 172.21.15.149:0/2818488005 shutdown_connections 2024-02-12T16:33:11.760 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:11.757+0000 7f3872503700 1 -- 172.21.15.149:0/2818488005 wait complete. 2024-02-12T16:33:11.760 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:12.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:10.306999+0000 mgr.smithi104.ohwbhf (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:12.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:11.756745+0000 mon.smithi104 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.149:0/2818488005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:13.374 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:13.375 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:14.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:12.307340+0000 mgr.smithi104.ohwbhf (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:15.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.196+0000 7fcd31789700 1 -- 172.21.15.149:0/3461775800 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c0f47c0 msgr2=0x7fcd2c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:15.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.196+0000 7fcd31789700 1 --2- 172.21.15.149:0/3461775800 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c0f47c0 0x7fcd2c0f4be0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fcd1c009fb0 tx=0x7fcd1c009b30).stop 2024-02-12T16:33:15.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.196+0000 7fcd31789700 1 -- 172.21.15.149:0/3461775800 shutdown_connections 2024-02-12T16:33:15.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.196+0000 7fcd31789700 1 --2- 172.21.15.149:0/3461775800 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c0f47c0 0x7fcd2c0f4be0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:15.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.196+0000 7fcd31789700 1 -- 172.21.15.149:0/3461775800 >> 172.21.15.149:0/3461775800 conn(0x7fcd2c0f02e0 msgr2=0x7fcd2c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:15.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.196+0000 7fcd31789700 1 -- 172.21.15.149:0/3461775800 shutdown_connections 2024-02-12T16:33:15.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.196+0000 7fcd31789700 1 -- 172.21.15.149:0/3461775800 wait complete. 2024-02-12T16:33:15.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.197+0000 7fcd31789700 1 Processor -- start 2024-02-12T16:33:15.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.197+0000 7fcd31789700 1 -- start start 2024-02-12T16:33:15.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.197+0000 7fcd31789700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c0f7330 0x7fcd2c0f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:15.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.197+0000 7fcd31789700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0f7750 con 0x7fcd2c0f7330 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.197+0000 7fcd2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c0f7330 0x7fcd2c0f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.197+0000 7fcd2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c0f7330 0x7fcd2c0f57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:38426/0 (socket says 172.21.15.149:38426) 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.197+0000 7fcd2affd700 1 -- 172.21.15.149:0/2989059037 learned_addr learned my addr 172.21.15.149:0/2989059037 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.198+0000 7fcd2affd700 1 -- 172.21.15.149:0/2989059037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c005040 con 0x7fcd2c0f7330 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.198+0000 7fcd2affd700 1 --2- 172.21.15.149:0/2989059037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c0f7330 0x7fcd2c0f57f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fcd1c009f80 tx=0x7fcd1c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.198+0000 7fcd137fe700 1 -- 172.21.15.149:0/2989059037 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd1c013380 con 0x7fcd2c0f7330 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.198+0000 7fcd31789700 1 -- 172.21.15.149:0/2989059037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c0f5ce0 con 0x7fcd2c0f7330 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.198+0000 7fcd137fe700 1 -- 172.21.15.149:0/2989059037 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd1c013790 con 0x7fcd2c0f7330 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.198+0000 7fcd137fe700 1 -- 172.21.15.149:0/2989059037 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd1c014420 con 0x7fcd2c0f7330 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.198+0000 7fcd31789700 1 -- 172.21.15.149:0/2989059037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c0f61d0 con 0x7fcd2c0f7330 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.199+0000 7fcd137fe700 1 -- 172.21.15.149:0/2989059037 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd1c025060 con 0x7fcd2c0f7330 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.199+0000 7fcd137fe700 1 --2- 172.21.15.149:0/2989059037 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd1402ff60 0x7fcd14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:15.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.199+0000 7fcd31789700 1 -- 172.21.15.149:0/2989059037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2c045ad0 con 0x7fcd2c0f7330 2024-02-12T16:33:15.204 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.199+0000 7fcd137fe700 1 -- 172.21.15.149:0/2989059037 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd1c040d50 con 0x7fcd2c0f7330 2024-02-12T16:33:15.204 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.199+0000 7fcd2a7fc700 1 --2- 172.21.15.149:0/2989059037 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd1402ff60 0x7fcd14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:15.204 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.200+0000 7fcd2a7fc700 1 --2- 172.21.15.149:0/2989059037 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd1402ff60 0x7fcd14032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcd200096a0 tx=0x7fcd20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:15.206 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.204+0000 7fcd137fe700 1 -- 172.21.15.149:0/2989059037 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd1c00f280 con 0x7fcd2c0f7330 2024-02-12T16:33:15.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.409+0000 7fcd31789700 1 -- 172.21.15.149:0/2989059037 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd2c10ca80 con 0x7fcd2c0f7330 2024-02-12T16:33:15.412 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.410+0000 7fcd137fe700 1 -- 172.21.15.149:0/2989059037 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd1c029020 con 0x7fcd2c0f7330 2024-02-12T16:33:15.412 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:15.412 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:15.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.411+0000 7fcd31789700 1 -- 172.21.15.149:0/2989059037 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd1402ff60 msgr2=0x7fcd14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:15.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.411+0000 7fcd31789700 1 --2- 172.21.15.149:0/2989059037 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd1402ff60 0x7fcd14032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcd200096a0 tx=0x7fcd20006b40).stop 2024-02-12T16:33:15.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.411+0000 7fcd31789700 1 -- 172.21.15.149:0/2989059037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c0f7330 msgr2=0x7fcd2c0f57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:15.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.411+0000 7fcd31789700 1 --2- 172.21.15.149:0/2989059037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c0f7330 0x7fcd2c0f57f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fcd1c009f80 tx=0x7fcd1c009b30).stop 2024-02-12T16:33:15.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.411+0000 7fcd31789700 1 -- 172.21.15.149:0/2989059037 shutdown_connections 2024-02-12T16:33:15.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.411+0000 7fcd31789700 1 --2- 172.21.15.149:0/2989059037 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd1402ff60 0x7fcd14032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:15.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.411+0000 7fcd31789700 1 --2- 172.21.15.149:0/2989059037 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd2c0f7330 0x7fcd2c0f57f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:15.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.411+0000 7fcd31789700 1 -- 172.21.15.149:0/2989059037 >> 172.21.15.149:0/2989059037 conn(0x7fcd2c0f02e0 msgr2=0x7fcd2c0f0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:15.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.411+0000 7fcd31789700 1 -- 172.21.15.149:0/2989059037 shutdown_connections 2024-02-12T16:33:15.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:15.411+0000 7fcd31789700 1 -- 172.21.15.149:0/2989059037 wait complete. 2024-02-12T16:33:15.415 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:16.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:14.307606+0000 mgr.smithi104.ohwbhf (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:16.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:15.410774+0000 mon.smithi104 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.149:0/2989059037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:17.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:17.061 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:18.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:16.307974+0000 mgr.smithi104.ohwbhf (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:18.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.875+0000 7ffb4ffff700 1 -- 172.21.15.149:0/4110332453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb500f51a0 msgr2=0x7ffb500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:18.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.875+0000 7ffb4ffff700 1 --2- 172.21.15.149:0/4110332453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb500f51a0 0x7ffb500f55c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7ffb40009fb0 tx=0x7ffb40009b30).stop 2024-02-12T16:33:18.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.875+0000 7ffb4ffff700 1 -- 172.21.15.149:0/4110332453 shutdown_connections 2024-02-12T16:33:18.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.875+0000 7ffb4ffff700 1 --2- 172.21.15.149:0/4110332453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb500f51a0 0x7ffb500f55c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:18.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.875+0000 7ffb4ffff700 1 -- 172.21.15.149:0/4110332453 >> 172.21.15.149:0/4110332453 conn(0x7ffb500f09a0 msgr2=0x7ffb500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:18.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.875+0000 7ffb4ffff700 1 -- 172.21.15.149:0/4110332453 shutdown_connections 2024-02-12T16:33:18.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.875+0000 7ffb4ffff700 1 -- 172.21.15.149:0/4110332453 wait complete. 2024-02-12T16:33:18.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.875+0000 7ffb4ffff700 1 Processor -- start 2024-02-12T16:33:18.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.875+0000 7ffb4ffff700 1 -- start start 2024-02-12T16:33:18.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb4ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb500f51a0 0x7ffb501065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:18.880 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb4ffff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb50106ac0 con 0x7ffb500f51a0 2024-02-12T16:33:18.880 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb4effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb500f51a0 0x7ffb501065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:18.880 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb4effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb500f51a0 0x7ffb501065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:38444/0 (socket says 172.21.15.149:38444) 2024-02-12T16:33:18.880 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb4effd700 1 -- 172.21.15.149:0/2982801975 learned_addr learned my addr 172.21.15.149:0/2982801975 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:18.880 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb4effd700 1 -- 172.21.15.149:0/2982801975 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb40005040 con 0x7ffb500f51a0 2024-02-12T16:33:18.880 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb4effd700 1 --2- 172.21.15.149:0/2982801975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb500f51a0 0x7ffb501065d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ffb400089f0 tx=0x7ffb4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:18.880 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb377fe700 1 -- 172.21.15.149:0/2982801975 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb40009320 con 0x7ffb500f51a0 2024-02-12T16:33:18.880 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb4ffff700 1 -- 172.21.15.149:0/2982801975 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb50106cc0 con 0x7ffb500f51a0 2024-02-12T16:33:18.880 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb377fe700 1 -- 172.21.15.149:0/2982801975 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb40013380 con 0x7ffb500f51a0 2024-02-12T16:33:18.880 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb377fe700 1 -- 172.21.15.149:0/2982801975 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb40014e20 con 0x7ffb500f51a0 2024-02-12T16:33:18.880 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.876+0000 7ffb4ffff700 1 -- 172.21.15.149:0/2982801975 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb50107150 con 0x7ffb500f51a0 2024-02-12T16:33:18.881 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.877+0000 7ffb377fe700 1 -- 172.21.15.149:0/2982801975 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb400095f0 con 0x7ffb500f51a0 2024-02-12T16:33:18.881 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.877+0000 7ffb4ffff700 1 -- 172.21.15.149:0/2982801975 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb50045ad0 con 0x7ffb500f51a0 2024-02-12T16:33:18.881 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.877+0000 7ffb377fe700 1 --2- 172.21.15.149:0/2982801975 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb3802ff10 0x7ffb380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:18.882 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.877+0000 7ffb377fe700 1 -- 172.21.15.149:0/2982801975 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb40040e90 con 0x7ffb500f51a0 2024-02-12T16:33:18.882 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.878+0000 7ffb4e7fc700 1 --2- 172.21.15.149:0/2982801975 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb3802ff10 0x7ffb380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:18.882 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.878+0000 7ffb4e7fc700 1 --2- 172.21.15.149:0/2982801975 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb3802ff10 0x7ffb380323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ffb440096a0 tx=0x7ffb44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:18.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:18.882+0000 7ffb377fe700 1 -- 172.21.15.149:0/2982801975 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb40014420 con 0x7ffb500f51a0 2024-02-12T16:33:19.089 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.087+0000 7ffb4ffff700 1 -- 172.21.15.149:0/2982801975 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb500fad30 con 0x7ffb500f51a0 2024-02-12T16:33:19.090 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.087+0000 7ffb377fe700 1 -- 172.21.15.149:0/2982801975 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffb4001d980 con 0x7ffb500f51a0 2024-02-12T16:33:19.090 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:19.090 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:19.091 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.089+0000 7ffb4ffff700 1 -- 172.21.15.149:0/2982801975 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb3802ff10 msgr2=0x7ffb380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:19.091 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.089+0000 7ffb4ffff700 1 --2- 172.21.15.149:0/2982801975 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb3802ff10 0x7ffb380323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ffb440096a0 tx=0x7ffb44006b40).stop 2024-02-12T16:33:19.091 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.089+0000 7ffb4ffff700 1 -- 172.21.15.149:0/2982801975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb500f51a0 msgr2=0x7ffb501065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:19.091 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.089+0000 7ffb4ffff700 1 --2- 172.21.15.149:0/2982801975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb500f51a0 0x7ffb501065d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ffb400089f0 tx=0x7ffb4000b040).stop 2024-02-12T16:33:19.092 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.089+0000 7ffb4ffff700 1 -- 172.21.15.149:0/2982801975 shutdown_connections 2024-02-12T16:33:19.092 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.089+0000 7ffb4ffff700 1 --2- 172.21.15.149:0/2982801975 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ffb3802ff10 0x7ffb380323d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:19.092 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.089+0000 7ffb4ffff700 1 --2- 172.21.15.149:0/2982801975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb500f51a0 0x7ffb501065d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:19.092 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.089+0000 7ffb4ffff700 1 -- 172.21.15.149:0/2982801975 >> 172.21.15.149:0/2982801975 conn(0x7ffb500f09a0 msgr2=0x7ffb5005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:19.092 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.089+0000 7ffb4ffff700 1 -- 172.21.15.149:0/2982801975 shutdown_connections 2024-02-12T16:33:19.092 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:19.089+0000 7ffb4ffff700 1 -- 172.21.15.149:0/2982801975 wait complete. 2024-02-12T16:33:19.092 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:19.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:19 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:19.088851+0000 mon.smithi104 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.149:0/2982801975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:20.667 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:20.668 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:18.308314+0000 mgr.smithi104.ohwbhf (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.359+0000 7f6cdef4d700 1 -- 172.21.15.149:0/955653940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cd8060220 msgr2=0x7f6cd80f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.359+0000 7f6cdef4d700 1 --2- 172.21.15.149:0/955653940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cd8060220 0x7f6cd80f9150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f6ccc009fb0 tx=0x7f6ccc009b30).stop 2024-02-12T16:33:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.359+0000 7f6cdef4d700 1 -- 172.21.15.149:0/955653940 shutdown_connections 2024-02-12T16:33:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.359+0000 7f6cdef4d700 1 --2- 172.21.15.149:0/955653940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cd8060220 0x7f6cd80f9150 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.359+0000 7f6cdef4d700 1 -- 172.21.15.149:0/955653940 >> 172.21.15.149:0/955653940 conn(0x7f6cd80f0b70 msgr2=0x7f6cd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.359+0000 7f6cdef4d700 1 -- 172.21.15.149:0/955653940 shutdown_connections 2024-02-12T16:33:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.359+0000 7f6cdef4d700 1 -- 172.21.15.149:0/955653940 wait complete. 2024-02-12T16:33:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.359+0000 7f6cdef4d700 1 Processor -- start 2024-02-12T16:33:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.360+0000 7f6cdef4d700 1 -- start start 2024-02-12T16:33:22.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.360+0000 7f6cdef4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cd8060220 0x7f6cd81089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:22.363 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.360+0000 7f6cdef4d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd8108e90 con 0x7f6cd8060220 2024-02-12T16:33:22.364 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.360+0000 7f6cdcce9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cd8060220 0x7f6cd81089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:22.364 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.360+0000 7f6cdcce9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cd8060220 0x7f6cd81089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:38456/0 (socket says 172.21.15.149:38456) 2024-02-12T16:33:22.364 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.360+0000 7f6cdcce9700 1 -- 172.21.15.149:0/1566465434 learned_addr learned my addr 172.21.15.149:0/1566465434 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:22.364 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.361+0000 7f6cdcce9700 1 -- 172.21.15.149:0/1566465434 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc005040 con 0x7f6cd8060220 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.361+0000 7f6cdcce9700 1 --2- 172.21.15.149:0/1566465434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cd8060220 0x7f6cd81089a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f6ccc0089f0 tx=0x7f6ccc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.361+0000 7f6cd57fa700 1 -- 172.21.15.149:0/1566465434 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ccc009320 con 0x7f6cd8060220 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.361+0000 7f6cdef4d700 1 -- 172.21.15.149:0/1566465434 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd8109090 con 0x7f6cd8060220 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.361+0000 7f6cd57fa700 1 -- 172.21.15.149:0/1566465434 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ccc013380 con 0x7f6cd8060220 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.361+0000 7f6cd57fa700 1 -- 172.21.15.149:0/1566465434 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ccc014e20 con 0x7f6cd8060220 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.361+0000 7f6cdef4d700 1 -- 172.21.15.149:0/1566465434 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd8109520 con 0x7f6cd8060220 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.362+0000 7f6cd57fa700 1 -- 172.21.15.149:0/1566465434 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ccc014420 con 0x7f6cd8060220 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.362+0000 7f6cdef4d700 1 -- 172.21.15.149:0/1566465434 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd8045ad0 con 0x7f6cd8060220 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.362+0000 7f6cd57fa700 1 --2- 172.21.15.149:0/1566465434 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6cc802ff60 0x7f6cc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.362+0000 7f6cd57fa700 1 -- 172.21.15.149:0/1566465434 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ccc040fc0 con 0x7f6cd8060220 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.362+0000 7f6cd7fff700 1 --2- 172.21.15.149:0/1566465434 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6cc802ff60 0x7f6cc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:22.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.362+0000 7f6cd7fff700 1 --2- 172.21.15.149:0/1566465434 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6cc802ff60 0x7f6cc8032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f6cc40096a0 tx=0x7f6cc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:22.368 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.367+0000 7f6cd57fa700 1 -- 172.21.15.149:0/1566465434 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ccc01db70 con 0x7f6cd8060220 2024-02-12T16:33:22.571 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.569+0000 7f6cdef4d700 1 -- 172.21.15.149:0/1566465434 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cd810a3d0 con 0x7f6cd8060220 2024-02-12T16:33:22.571 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.570+0000 7f6cd57fa700 1 -- 172.21.15.149:0/1566465434 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6ccc01db70 con 0x7f6cd8060220 2024-02-12T16:33:22.572 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:22.572 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:22.573 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.571+0000 7f6cdef4d700 1 -- 172.21.15.149:0/1566465434 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6cc802ff60 msgr2=0x7f6cc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:22.574 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.571+0000 7f6cdef4d700 1 --2- 172.21.15.149:0/1566465434 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6cc802ff60 0x7f6cc8032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f6cc40096a0 tx=0x7f6cc4006b40).stop 2024-02-12T16:33:22.574 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.571+0000 7f6cdef4d700 1 -- 172.21.15.149:0/1566465434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cd8060220 msgr2=0x7f6cd81089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:22.574 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.571+0000 7f6cdef4d700 1 --2- 172.21.15.149:0/1566465434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cd8060220 0x7f6cd81089a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f6ccc0089f0 tx=0x7f6ccc00b040).stop 2024-02-12T16:33:22.574 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.571+0000 7f6cdef4d700 1 -- 172.21.15.149:0/1566465434 shutdown_connections 2024-02-12T16:33:22.574 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.571+0000 7f6cdef4d700 1 --2- 172.21.15.149:0/1566465434 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6cc802ff60 0x7f6cc8032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:22.574 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.571+0000 7f6cdef4d700 1 --2- 172.21.15.149:0/1566465434 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cd8060220 0x7f6cd81089a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:22.574 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.571+0000 7f6cdef4d700 1 -- 172.21.15.149:0/1566465434 >> 172.21.15.149:0/1566465434 conn(0x7f6cd80f0b70 msgr2=0x7f6cd80f15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:22.574 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.571+0000 7f6cdef4d700 1 -- 172.21.15.149:0/1566465434 shutdown_connections 2024-02-12T16:33:22.574 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:22.571+0000 7f6cdef4d700 1 -- 172.21.15.149:0/1566465434 wait complete. 2024-02-12T16:33:22.575 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:22.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:20.308837+0000 mgr.smithi104.ohwbhf (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:23.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:23 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:22.570876+0000 mon.smithi104 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.149:0/1566465434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:24.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:24.192 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:22.309182+0000 mgr.smithi104.ohwbhf (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:24.405673+0000 mon.smithi104 (mon.0) 178 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:33:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:24.412426+0000 mon.smithi104 (mon.0) 179 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:33:26.135 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.131+0000 7f12de0c3700 1 -- 172.21.15.149:0/3045761955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d8060200 msgr2=0x7f12d80fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:26.135 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.131+0000 7f12de0c3700 1 --2- 172.21.15.149:0/3045761955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d8060200 0x7f12d80fb570 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f12c0009fb0 tx=0x7f12c0009b30).stop 2024-02-12T16:33:26.135 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.131+0000 7f12de0c3700 1 -- 172.21.15.149:0/3045761955 shutdown_connections 2024-02-12T16:33:26.135 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.131+0000 7f12de0c3700 1 --2- 172.21.15.149:0/3045761955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d8060200 0x7f12d80fb570 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.131+0000 7f12de0c3700 1 -- 172.21.15.149:0/3045761955 >> 172.21.15.149:0/3045761955 conn(0x7f12d80f0b70 msgr2=0x7f12d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.131+0000 7f12de0c3700 1 -- 172.21.15.149:0/3045761955 shutdown_connections 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.131+0000 7f12de0c3700 1 -- 172.21.15.149:0/3045761955 wait complete. 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.132+0000 7f12de0c3700 1 Processor -- start 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.132+0000 7f12de0c3700 1 -- start start 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.132+0000 7f12de0c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d8060200 0x7f12d81089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.132+0000 7f12de0c3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d8108e90 con 0x7f12d8060200 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.132+0000 7f12d77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d8060200 0x7f12d81089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.133+0000 7f12d77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d8060200 0x7f12d81089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:54220/0 (socket says 172.21.15.149:54220) 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.133+0000 7f12d77fe700 1 -- 172.21.15.149:0/3824874698 learned_addr learned my addr 172.21.15.149:0/3824874698 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.133+0000 7f12d77fe700 1 -- 172.21.15.149:0/3824874698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c0005040 con 0x7f12d8060200 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.133+0000 7f12d77fe700 1 --2- 172.21.15.149:0/3824874698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d8060200 0x7f12d81089a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f12c00089f0 tx=0x7f12c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:26.136 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.133+0000 7f12cffff700 1 -- 172.21.15.149:0/3824874698 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c0009320 con 0x7f12d8060200 2024-02-12T16:33:26.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.133+0000 7f12de0c3700 1 -- 172.21.15.149:0/3824874698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d8109090 con 0x7f12d8060200 2024-02-12T16:33:26.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.133+0000 7f12cffff700 1 -- 172.21.15.149:0/3824874698 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12c0013380 con 0x7f12d8060200 2024-02-12T16:33:26.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.133+0000 7f12cffff700 1 -- 172.21.15.149:0/3824874698 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c0014e20 con 0x7f12d8060200 2024-02-12T16:33:26.137 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.133+0000 7f12de0c3700 1 -- 172.21.15.149:0/3824874698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d8109520 con 0x7f12d8060200 2024-02-12T16:33:26.138 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.134+0000 7f12cffff700 1 -- 172.21.15.149:0/3824874698 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12c0014420 con 0x7f12d8060200 2024-02-12T16:33:26.138 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.134+0000 7f12de0c3700 1 -- 172.21.15.149:0/3824874698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d8045ad0 con 0x7f12d8060200 2024-02-12T16:33:26.138 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.134+0000 7f12cffff700 1 --2- 172.21.15.149:0/3824874698 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12c402ff10 0x7f12c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:26.138 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.134+0000 7f12cffff700 1 -- 172.21.15.149:0/3824874698 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12c0040f50 con 0x7f12d8060200 2024-02-12T16:33:26.138 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.134+0000 7f12d6ffd700 1 --2- 172.21.15.149:0/3824874698 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12c402ff10 0x7f12c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:26.138 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.135+0000 7f12d6ffd700 1 --2- 172.21.15.149:0/3824874698 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12c402ff10 0x7f12c40323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f12c80096a0 tx=0x7f12c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:26.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.140+0000 7f12cffff700 1 -- 172.21.15.149:0/3824874698 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12c006d2b0 con 0x7f12d8060200 2024-02-12T16:33:26.348 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.346+0000 7f12de0c3700 1 -- 172.21.15.149:0/3824874698 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12d810a3d0 con 0x7f12d8060200 2024-02-12T16:33:26.349 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.347+0000 7f12cffff700 1 -- 172.21.15.149:0/3824874698 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f12c001dd50 con 0x7f12d8060200 2024-02-12T16:33:26.349 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:26.349 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:26.351 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.349+0000 7f12de0c3700 1 -- 172.21.15.149:0/3824874698 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12c402ff10 msgr2=0x7f12c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:26.351 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.349+0000 7f12de0c3700 1 --2- 172.21.15.149:0/3824874698 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12c402ff10 0x7f12c40323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f12c80096a0 tx=0x7f12c8006b40).stop 2024-02-12T16:33:26.351 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.349+0000 7f12de0c3700 1 -- 172.21.15.149:0/3824874698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d8060200 msgr2=0x7f12d81089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:26.351 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.349+0000 7f12de0c3700 1 --2- 172.21.15.149:0/3824874698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d8060200 0x7f12d81089a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f12c00089f0 tx=0x7f12c000b040).stop 2024-02-12T16:33:26.351 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.349+0000 7f12de0c3700 1 -- 172.21.15.149:0/3824874698 shutdown_connections 2024-02-12T16:33:26.351 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.349+0000 7f12de0c3700 1 --2- 172.21.15.149:0/3824874698 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12c402ff10 0x7f12c40323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:26.351 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.349+0000 7f12de0c3700 1 --2- 172.21.15.149:0/3824874698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12d8060200 0x7f12d81089a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:26.351 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.349+0000 7f12de0c3700 1 -- 172.21.15.149:0/3824874698 >> 172.21.15.149:0/3824874698 conn(0x7f12d80f0b70 msgr2=0x7f12d80f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:26.351 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.349+0000 7f12de0c3700 1 -- 172.21.15.149:0/3824874698 shutdown_connections 2024-02-12T16:33:26.351 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:26.349+0000 7f12de0c3700 1 -- 172.21.15.149:0/3824874698 wait complete. 2024-02-12T16:33:26.352 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:26.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:24.309436+0000 mgr.smithi104.ohwbhf (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:27.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:26.348318+0000 mon.smithi104 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.149:0/3824874698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:27.980 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:27.980 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:28.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:26.309782+0000 mgr.smithi104.ohwbhf (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:29.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.787+0000 7f7f84dfa700 1 -- 172.21.15.149:0/2681323116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f800f62a0 msgr2=0x7f7f800f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:29.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.787+0000 7f7f84dfa700 1 --2- 172.21.15.149:0/2681323116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f800f62a0 0x7f7f800f66c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f7f70009fb0 tx=0x7f7f70009b30).stop 2024-02-12T16:33:29.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.787+0000 7f7f84dfa700 1 -- 172.21.15.149:0/2681323116 shutdown_connections 2024-02-12T16:33:29.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.787+0000 7f7f84dfa700 1 --2- 172.21.15.149:0/2681323116 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f800f62a0 0x7f7f800f66c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:29.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.787+0000 7f7f84dfa700 1 -- 172.21.15.149:0/2681323116 >> 172.21.15.149:0/2681323116 conn(0x7f7f800f1ac0 msgr2=0x7f7f800f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:29.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.787+0000 7f7f84dfa700 1 -- 172.21.15.149:0/2681323116 shutdown_connections 2024-02-12T16:33:29.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.787+0000 7f7f84dfa700 1 -- 172.21.15.149:0/2681323116 wait complete. 2024-02-12T16:33:29.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.787+0000 7f7f84dfa700 1 Processor -- start 2024-02-12T16:33:29.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.788+0000 7f7f84dfa700 1 -- start start 2024-02-12T16:33:29.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.788+0000 7f7f84dfa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f800f62a0 0x7f7f8006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:29.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.788+0000 7f7f84dfa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f80069e20 con 0x7f7f800f62a0 2024-02-12T16:33:29.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.788+0000 7f7f7f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f800f62a0 0x7f7f8006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:29.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.788+0000 7f7f7f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f800f62a0 0x7f7f8006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:54240/0 (socket says 172.21.15.149:54240) 2024-02-12T16:33:29.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.788+0000 7f7f7f7fe700 1 -- 172.21.15.149:0/994198620 learned_addr learned my addr 172.21.15.149:0/994198620 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:29.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.788+0000 7f7f7f7fe700 1 -- 172.21.15.149:0/994198620 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f70005040 con 0x7f7f800f62a0 2024-02-12T16:33:29.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.788+0000 7f7f7f7fe700 1 --2- 172.21.15.149:0/994198620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f800f62a0 0x7f7f8006b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f7f700089f0 tx=0x7f7f7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:29.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.789+0000 7f7f67fff700 1 -- 172.21.15.149:0/994198620 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f70009320 con 0x7f7f800f62a0 2024-02-12T16:33:29.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.789+0000 7f7f84dfa700 1 -- 172.21.15.149:0/994198620 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f80069fc0 con 0x7f7f800f62a0 2024-02-12T16:33:29.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.789+0000 7f7f67fff700 1 -- 172.21.15.149:0/994198620 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f70013380 con 0x7f7f800f62a0 2024-02-12T16:33:29.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.789+0000 7f7f67fff700 1 -- 172.21.15.149:0/994198620 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f70014e20 con 0x7f7f800f62a0 2024-02-12T16:33:29.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.789+0000 7f7f84dfa700 1 -- 172.21.15.149:0/994198620 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8006a450 con 0x7f7f800f62a0 2024-02-12T16:33:29.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.789+0000 7f7f67fff700 1 -- 172.21.15.149:0/994198620 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f700263a0 con 0x7f7f800f62a0 2024-02-12T16:33:29.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.790+0000 7f7f67fff700 1 --2- 172.21.15.149:0/994198620 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7f6802ff10 0x7f7f680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:29.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.790+0000 7f7f67fff700 1 -- 172.21.15.149:0/994198620 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f70041010 con 0x7f7f800f62a0 2024-02-12T16:33:29.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.790+0000 7f7f84dfa700 1 -- 172.21.15.149:0/994198620 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f80059100 con 0x7f7f800f62a0 2024-02-12T16:33:29.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.790+0000 7f7f7effd700 1 --2- 172.21.15.149:0/994198620 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7f6802ff10 0x7f7f680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:29.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.790+0000 7f7f7effd700 1 --2- 172.21.15.149:0/994198620 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7f6802ff10 0x7f7f680323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f7f740096a0 tx=0x7f7f74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:29.795 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.794+0000 7f7f67fff700 1 -- 172.21.15.149:0/994198620 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f70012920 con 0x7f7f800f62a0 2024-02-12T16:33:30.000 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.999+0000 7f7f84dfa700 1 -- 172.21.15.149:0/994198620 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f80045ad0 con 0x7f7f800f62a0 2024-02-12T16:33:30.000 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:29.999+0000 7f7f67fff700 1 -- 172.21.15.149:0/994198620 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7f7002b020 con 0x7f7f800f62a0 2024-02-12T16:33:30.000 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:30.001 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:30.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:30.001+0000 7f7f84dfa700 1 -- 172.21.15.149:0/994198620 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7f6802ff10 msgr2=0x7f7f680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:30.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:30.001+0000 7f7f84dfa700 1 --2- 172.21.15.149:0/994198620 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7f6802ff10 0x7f7f680323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f7f740096a0 tx=0x7f7f74006b40).stop 2024-02-12T16:33:30.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:30.001+0000 7f7f84dfa700 1 -- 172.21.15.149:0/994198620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f800f62a0 msgr2=0x7f7f8006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:30.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:30.001+0000 7f7f84dfa700 1 --2- 172.21.15.149:0/994198620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f800f62a0 0x7f7f8006b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f7f700089f0 tx=0x7f7f7000b040).stop 2024-02-12T16:33:30.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:30.001+0000 7f7f84dfa700 1 -- 172.21.15.149:0/994198620 shutdown_connections 2024-02-12T16:33:30.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:30.001+0000 7f7f84dfa700 1 --2- 172.21.15.149:0/994198620 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7f6802ff10 0x7f7f680323d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:30.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:30.001+0000 7f7f84dfa700 1 --2- 172.21.15.149:0/994198620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f800f62a0 0x7f7f8006b9a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:30.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:30.001+0000 7f7f84dfa700 1 -- 172.21.15.149:0/994198620 >> 172.21.15.149:0/994198620 conn(0x7f7f800f1ac0 msgr2=0x7f7f800f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:30.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:30.001+0000 7f7f84dfa700 1 -- 172.21.15.149:0/994198620 shutdown_connections 2024-02-12T16:33:30.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:30.001+0000 7f7f84dfa700 1 -- 172.21.15.149:0/994198620 wait complete. 2024-02-12T16:33:30.003 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:30.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:28.310161+0000 mgr.smithi104.ohwbhf (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:30.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:29.999770+0000 mon.smithi104 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.149:0/994198620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:31.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:31.598 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:32.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:30.310478+0000 mgr.smithi104.ohwbhf (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:33.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.505+0000 7f6e21643700 1 -- 172.21.15.149:0/4240512162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e1c0f51a0 msgr2=0x7f6e1c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:33.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.505+0000 7f6e21643700 1 --2- 172.21.15.149:0/4240512162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e1c0f51a0 0x7f6e1c0f55c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f6e0c009fb0 tx=0x7f6e0c009b30).stop 2024-02-12T16:33:33.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.505+0000 7f6e21643700 1 -- 172.21.15.149:0/4240512162 shutdown_connections 2024-02-12T16:33:33.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.505+0000 7f6e21643700 1 --2- 172.21.15.149:0/4240512162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e1c0f51a0 0x7f6e1c0f55c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:33.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.505+0000 7f6e21643700 1 -- 172.21.15.149:0/4240512162 >> 172.21.15.149:0/4240512162 conn(0x7f6e1c0f09a0 msgr2=0x7f6e1c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.505+0000 7f6e21643700 1 -- 172.21.15.149:0/4240512162 shutdown_connections 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.506+0000 7f6e21643700 1 -- 172.21.15.149:0/4240512162 wait complete. 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.506+0000 7f6e21643700 1 Processor -- start 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.506+0000 7f6e21643700 1 -- start start 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.506+0000 7f6e21643700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e1c0fd2b0 0x7f6e1c0fb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.506+0000 7f6e21643700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0fd6d0 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.506+0000 7f6e1bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e1c0fd2b0 0x7f6e1c0fb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.507+0000 7f6e1bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e1c0fd2b0 0x7f6e1c0fb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:54256/0 (socket says 172.21.15.149:54256) 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.507+0000 7f6e1bfff700 1 -- 172.21.15.149:0/2996006755 learned_addr learned my addr 172.21.15.149:0/2996006755 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.507+0000 7f6e1bfff700 1 -- 172.21.15.149:0/2996006755 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c005040 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.507+0000 7f6e1bfff700 1 --2- 172.21.15.149:0/2996006755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e1c0fd2b0 0x7f6e1c0fb770 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f6e0c011000 tx=0x7f6e0c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.507+0000 7f6e18ff9700 1 -- 172.21.15.149:0/2996006755 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e0c013380 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.507+0000 7f6e18ff9700 1 -- 172.21.15.149:0/2996006755 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e0c013790 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.507+0000 7f6e21643700 1 -- 172.21.15.149:0/2996006755 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1c0fbc60 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.507+0000 7f6e18ff9700 1 -- 172.21.15.149:0/2996006755 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e0c01ee70 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.510 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.507+0000 7f6e21643700 1 -- 172.21.15.149:0/2996006755 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1c0fc150 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.510 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.508+0000 7f6e18ff9700 1 -- 172.21.15.149:0/2996006755 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e0c025060 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.508+0000 7f6e18ff9700 1 --2- 172.21.15.149:0/2996006755 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6e0402ffb0 0x7f6e04032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:33.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.508+0000 7f6e18ff9700 1 -- 172.21.15.149:0/2996006755 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e0c040f30 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.508+0000 7f6e21643700 1 -- 172.21.15.149:0/2996006755 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1c059100 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.508+0000 7f6e1b7fe700 1 --2- 172.21.15.149:0/2996006755 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6e0402ffb0 0x7f6e04032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:33.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.509+0000 7f6e1b7fe700 1 --2- 172.21.15.149:0/2996006755 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6e0402ffb0 0x7f6e04032470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6e100096a0 tx=0x7f6e10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:33.514 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.513+0000 7f6e18ff9700 1 -- 172.21.15.149:0/2996006755 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e0c028440 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.718+0000 7f6e21643700 1 -- 172.21.15.149:0/2996006755 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e1c045ad0 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.718+0000 7f6e18ff9700 1 -- 172.21.15.149:0/2996006755 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6e0c024260 con 0x7f6e1c0fd2b0 2024-02-12T16:33:33.720 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:33.720 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:33.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.720+0000 7f6e21643700 1 -- 172.21.15.149:0/2996006755 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6e0402ffb0 msgr2=0x7f6e04032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:33.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.720+0000 7f6e21643700 1 --2- 172.21.15.149:0/2996006755 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6e0402ffb0 0x7f6e04032470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6e100096a0 tx=0x7f6e10006b40).stop 2024-02-12T16:33:33.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.720+0000 7f6e21643700 1 -- 172.21.15.149:0/2996006755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e1c0fd2b0 msgr2=0x7f6e1c0fb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:33.722 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.720+0000 7f6e21643700 1 --2- 172.21.15.149:0/2996006755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e1c0fd2b0 0x7f6e1c0fb770 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f6e0c011000 tx=0x7f6e0c009b30).stop 2024-02-12T16:33:33.722 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.720+0000 7f6e21643700 1 -- 172.21.15.149:0/2996006755 shutdown_connections 2024-02-12T16:33:33.722 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.720+0000 7f6e21643700 1 --2- 172.21.15.149:0/2996006755 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6e0402ffb0 0x7f6e04032470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:33.722 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.720+0000 7f6e21643700 1 --2- 172.21.15.149:0/2996006755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e1c0fd2b0 0x7f6e1c0fb770 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:33.722 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.720+0000 7f6e21643700 1 -- 172.21.15.149:0/2996006755 >> 172.21.15.149:0/2996006755 conn(0x7f6e1c0f09a0 msgr2=0x7f6e1c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:33.722 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.720+0000 7f6e21643700 1 -- 172.21.15.149:0/2996006755 shutdown_connections 2024-02-12T16:33:33.722 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:33.720+0000 7f6e21643700 1 -- 172.21.15.149:0/2996006755 wait complete. 2024-02-12T16:33:33.722 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:34.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:32.310824+0000 mgr.smithi104.ohwbhf (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:34.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:33.718643+0000 mon.smithi104 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.149:0/2996006755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:35.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:35.311 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:36.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:34.311198+0000 mgr.smithi104.ohwbhf (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:37.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.097+0000 7fafa6989700 1 -- 172.21.15.149:0/329138449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa00f5410 msgr2=0x7fafa00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:37.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.097+0000 7fafa6989700 1 --2- 172.21.15.149:0/329138449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa00f5410 0x7fafa00f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7faf94009fb0 tx=0x7faf94009b30).stop 2024-02-12T16:33:37.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.097+0000 7fafa6989700 1 -- 172.21.15.149:0/329138449 shutdown_connections 2024-02-12T16:33:37.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.097+0000 7fafa6989700 1 --2- 172.21.15.149:0/329138449 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa00f5410 0x7fafa00f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:37.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.097+0000 7fafa6989700 1 -- 172.21.15.149:0/329138449 >> 172.21.15.149:0/329138449 conn(0x7fafa00f0b70 msgr2=0x7fafa00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:37.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.097+0000 7fafa6989700 1 -- 172.21.15.149:0/329138449 shutdown_connections 2024-02-12T16:33:37.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.097+0000 7fafa6989700 1 -- 172.21.15.149:0/329138449 wait complete. 2024-02-12T16:33:37.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.098+0000 7fafa6989700 1 Processor -- start 2024-02-12T16:33:37.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.098+0000 7fafa6989700 1 -- start start 2024-02-12T16:33:37.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.098+0000 7fafa6989700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa00f5410 0x7fafa01094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:37.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.098+0000 7fafa6989700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa0060240 con 0x7fafa00f5410 2024-02-12T16:33:37.102 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.098+0000 7faf9ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa00f5410 0x7fafa01094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.098+0000 7faf9ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa00f5410 0x7fafa01094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:59302/0 (socket says 172.21.15.149:59302) 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.098+0000 7faf9ffff700 1 -- 172.21.15.149:0/3739015626 learned_addr learned my addr 172.21.15.149:0/3739015626 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.099+0000 7faf9ffff700 1 -- 172.21.15.149:0/3739015626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf94005040 con 0x7fafa00f5410 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.099+0000 7faf9ffff700 1 --2- 172.21.15.149:0/3739015626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa00f5410 0x7fafa01094e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7faf94005010 tx=0x7faf9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.099+0000 7faf9cff9700 1 -- 172.21.15.149:0/3739015626 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf94009320 con 0x7fafa00f5410 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.099+0000 7fafa6989700 1 -- 172.21.15.149:0/3739015626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa0060440 con 0x7fafa00f5410 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.099+0000 7faf9cff9700 1 -- 172.21.15.149:0/3739015626 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf94013380 con 0x7fafa00f5410 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.099+0000 7faf9cff9700 1 -- 172.21.15.149:0/3739015626 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf94014e50 con 0x7fafa00f5410 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.099+0000 7fafa6989700 1 -- 172.21.15.149:0/3739015626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa0109b90 con 0x7fafa00f5410 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.100+0000 7faf9cff9700 1 -- 172.21.15.149:0/3739015626 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf940095f0 con 0x7fafa00f5410 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.100+0000 7fafa6989700 1 -- 172.21.15.149:0/3739015626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa0045ad0 con 0x7fafa00f5410 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.100+0000 7faf9cff9700 1 --2- 172.21.15.149:0/3739015626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf8802ff10 0x7faf880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.100+0000 7faf9cff9700 1 -- 172.21.15.149:0/3739015626 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf94040e70 con 0x7fafa00f5410 2024-02-12T16:33:37.103 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.100+0000 7faf9f7fe700 1 --2- 172.21.15.149:0/3739015626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf8802ff10 0x7faf880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:37.104 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.101+0000 7faf9f7fe700 1 --2- 172.21.15.149:0/3739015626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf8802ff10 0x7faf880323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7faf900096a0 tx=0x7faf90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:37.106 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.105+0000 7faf9cff9700 1 -- 172.21.15.149:0/3739015626 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf94014420 con 0x7fafa00f5410 2024-02-12T16:33:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.308+0000 7fafa6989700 1 -- 172.21.15.149:0/3739015626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafa010a3a0 con 0x7fafa00f5410 2024-02-12T16:33:37.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.309+0000 7faf9cff9700 1 -- 172.21.15.149:0/3739015626 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faf9400eed0 con 0x7fafa00f5410 2024-02-12T16:33:37.311 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:37.311 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:37.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.311+0000 7fafa6989700 1 -- 172.21.15.149:0/3739015626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf8802ff10 msgr2=0x7faf880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:37.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.311+0000 7fafa6989700 1 --2- 172.21.15.149:0/3739015626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf8802ff10 0x7faf880323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7faf900096a0 tx=0x7faf90006b40).stop 2024-02-12T16:33:37.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.311+0000 7fafa6989700 1 -- 172.21.15.149:0/3739015626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa00f5410 msgr2=0x7fafa01094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:37.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.311+0000 7fafa6989700 1 --2- 172.21.15.149:0/3739015626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa00f5410 0x7fafa01094e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7faf94005010 tx=0x7faf9400b040).stop 2024-02-12T16:33:37.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.311+0000 7fafa6989700 1 -- 172.21.15.149:0/3739015626 shutdown_connections 2024-02-12T16:33:37.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.311+0000 7fafa6989700 1 --2- 172.21.15.149:0/3739015626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf8802ff10 0x7faf880323d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:37.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.311+0000 7fafa6989700 1 --2- 172.21.15.149:0/3739015626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafa00f5410 0x7fafa01094e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:37.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.311+0000 7fafa6989700 1 -- 172.21.15.149:0/3739015626 >> 172.21.15.149:0/3739015626 conn(0x7fafa00f0b70 msgr2=0x7fafa00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:37.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.311+0000 7fafa6989700 1 -- 172.21.15.149:0/3739015626 shutdown_connections 2024-02-12T16:33:37.314 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:37.311+0000 7fafa6989700 1 -- 172.21.15.149:0/3739015626 wait complete. 2024-02-12T16:33:37.314 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:38.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:36.311525+0000 mgr.smithi104.ohwbhf (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:38.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:37.310373+0000 mon.smithi104 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.149:0/3739015626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:38.959 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:38.960 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:40.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:38.311843+0000 mgr.smithi104.ohwbhf (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:40.788 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.785+0000 7f1fc7028700 1 -- 172.21.15.149:0/3467385731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc00f6510 msgr2=0x7f1fc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:40.788 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.785+0000 7f1fc7028700 1 --2- 172.21.15.149:0/3467385731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc00f6510 0x7f1fc00f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f1fb4009fb0 tx=0x7f1fb4009b30).stop 2024-02-12T16:33:40.788 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.785+0000 7f1fc7028700 1 -- 172.21.15.149:0/3467385731 shutdown_connections 2024-02-12T16:33:40.788 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.785+0000 7f1fc7028700 1 --2- 172.21.15.149:0/3467385731 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc00f6510 0x7f1fc00f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:40.788 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.785+0000 7f1fc7028700 1 -- 172.21.15.149:0/3467385731 >> 172.21.15.149:0/3467385731 conn(0x7f1fc00f1c70 msgr2=0x7f1fc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:40.788 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.785+0000 7f1fc7028700 1 -- 172.21.15.149:0/3467385731 shutdown_connections 2024-02-12T16:33:40.788 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.785+0000 7f1fc7028700 1 -- 172.21.15.149:0/3467385731 wait complete. 2024-02-12T16:33:40.788 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.786+0000 7f1fc7028700 1 Processor -- start 2024-02-12T16:33:40.788 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.786+0000 7f1fc7028700 1 -- start start 2024-02-12T16:33:40.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.786+0000 7f1fc7028700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc00f6510 0x7f1fc01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:40.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.786+0000 7f1fc7028700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc0106cd0 con 0x7f1fc00f6510 2024-02-12T16:33:40.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.787+0000 7f1fc4dc4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc00f6510 0x7f1fc01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:40.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.787+0000 7f1fc4dc4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc00f6510 0x7f1fc01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:59328/0 (socket says 172.21.15.149:59328) 2024-02-12T16:33:40.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.787+0000 7f1fc4dc4700 1 -- 172.21.15.149:0/749349838 learned_addr learned my addr 172.21.15.149:0/749349838 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:40.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.787+0000 7f1fc4dc4700 1 -- 172.21.15.149:0/749349838 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4005040 con 0x7f1fc00f6510 2024-02-12T16:33:40.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.787+0000 7f1fc4dc4700 1 --2- 172.21.15.149:0/749349838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc00f6510 0x7f1fc01067e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f1fb4005010 tx=0x7f1fb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:40.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.787+0000 7f1fbd7fa700 1 -- 172.21.15.149:0/749349838 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb4009320 con 0x7f1fc00f6510 2024-02-12T16:33:40.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.787+0000 7f1fc7028700 1 -- 172.21.15.149:0/749349838 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc0106ed0 con 0x7f1fc00f6510 2024-02-12T16:33:40.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.787+0000 7f1fbd7fa700 1 -- 172.21.15.149:0/749349838 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fb4013380 con 0x7f1fc00f6510 2024-02-12T16:33:40.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.787+0000 7f1fbd7fa700 1 -- 172.21.15.149:0/749349838 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb4014e50 con 0x7f1fc00f6510 2024-02-12T16:33:40.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.787+0000 7f1fc7028700 1 -- 172.21.15.149:0/749349838 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc0107360 con 0x7f1fc00f6510 2024-02-12T16:33:40.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.788+0000 7f1fbd7fa700 1 -- 172.21.15.149:0/749349838 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fb4014420 con 0x7f1fc00f6510 2024-02-12T16:33:40.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.788+0000 7f1fc7028700 1 -- 172.21.15.149:0/749349838 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc0045ad0 con 0x7f1fc00f6510 2024-02-12T16:33:40.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.788+0000 7f1fbd7fa700 1 --2- 172.21.15.149:0/749349838 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1fb002ff10 0x7f1fb00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:40.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.788+0000 7f1fbd7fa700 1 -- 172.21.15.149:0/749349838 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fb4040f70 con 0x7f1fc00f6510 2024-02-12T16:33:40.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.789+0000 7f1fbffff700 1 --2- 172.21.15.149:0/749349838 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1fb002ff10 0x7f1fb00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:40.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.789+0000 7f1fbffff700 1 --2- 172.21.15.149:0/749349838 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1fb002ff10 0x7f1fb00323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f1fac0096a0 tx=0x7f1fac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:40.795 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.793+0000 7f1fbd7fa700 1 -- 172.21.15.149:0/749349838 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fb40095f0 con 0x7f1fc00f6510 2024-02-12T16:33:40.998 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.996+0000 7f1fc7028700 1 -- 172.21.15.149:0/749349838 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fc0108210 con 0x7f1fc00f6510 2024-02-12T16:33:40.999 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.997+0000 7f1fbd7fa700 1 -- 172.21.15.149:0/749349838 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1fb401db70 con 0x7f1fc00f6510 2024-02-12T16:33:40.999 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:41.000 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:41.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.998+0000 7f1fc7028700 1 -- 172.21.15.149:0/749349838 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1fb002ff10 msgr2=0x7f1fb00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:41.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.998+0000 7f1fc7028700 1 --2- 172.21.15.149:0/749349838 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1fb002ff10 0x7f1fb00323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f1fac0096a0 tx=0x7f1fac006b40).stop 2024-02-12T16:33:41.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.998+0000 7f1fc7028700 1 -- 172.21.15.149:0/749349838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc00f6510 msgr2=0x7f1fc01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:41.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.998+0000 7f1fc7028700 1 --2- 172.21.15.149:0/749349838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc00f6510 0x7f1fc01067e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f1fb4005010 tx=0x7f1fb400b040).stop 2024-02-12T16:33:41.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.998+0000 7f1fc7028700 1 -- 172.21.15.149:0/749349838 shutdown_connections 2024-02-12T16:33:41.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.998+0000 7f1fc7028700 1 --2- 172.21.15.149:0/749349838 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1fb002ff10 0x7f1fb00323d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:41.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.998+0000 7f1fc7028700 1 --2- 172.21.15.149:0/749349838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fc00f6510 0x7f1fc01067e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:41.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.998+0000 7f1fc7028700 1 -- 172.21.15.149:0/749349838 >> 172.21.15.149:0/749349838 conn(0x7f1fc00f1c70 msgr2=0x7f1fc00f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:41.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.998+0000 7f1fc7028700 1 -- 172.21.15.149:0/749349838 shutdown_connections 2024-02-12T16:33:41.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:40.999+0000 7f1fc7028700 1 -- 172.21.15.149:0/749349838 wait complete. 2024-02-12T16:33:41.003 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:41.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:41 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:40.998151+0000 mon.smithi104 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.149:0/749349838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:42.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:40.312200+0000 mgr.smithi104.ohwbhf (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:42.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:42.713 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:44.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.528+0000 7f12c65fa700 1 -- 172.21.15.149:0/557597951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c00f53f0 msgr2=0x7f12c00f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:44.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.528+0000 7f12c65fa700 1 --2- 172.21.15.149:0/557597951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c00f53f0 0x7f12c00f5810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f12b0009fb0 tx=0x7f12b0009b30).stop 2024-02-12T16:33:44.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.528+0000 7f12c65fa700 1 -- 172.21.15.149:0/557597951 shutdown_connections 2024-02-12T16:33:44.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.528+0000 7f12c65fa700 1 --2- 172.21.15.149:0/557597951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c00f53f0 0x7f12c00f5810 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:44.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.528+0000 7f12c65fa700 1 -- 172.21.15.149:0/557597951 >> 172.21.15.149:0/557597951 conn(0x7f12c00f0b70 msgr2=0x7f12c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:44.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.528+0000 7f12c65fa700 1 -- 172.21.15.149:0/557597951 shutdown_connections 2024-02-12T16:33:44.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.528+0000 7f12c65fa700 1 -- 172.21.15.149:0/557597951 wait complete. 2024-02-12T16:33:44.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.529+0000 7f12c65fa700 1 Processor -- start 2024-02-12T16:33:44.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.529+0000 7f12c65fa700 1 -- start start 2024-02-12T16:33:44.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.529+0000 7f12c65fa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c00f53f0 0x7f12c0109480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:44.532 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.529+0000 7f12c65fa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0060280 con 0x7f12c00f53f0 2024-02-12T16:33:44.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.529+0000 7f12bffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c00f53f0 0x7f12c0109480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:44.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.529+0000 7f12bffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c00f53f0 0x7f12c0109480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:57648/0 (socket says 172.21.15.149:57648) 2024-02-12T16:33:44.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.529+0000 7f12bffff700 1 -- 172.21.15.149:0/4004181110 learned_addr learned my addr 172.21.15.149:0/4004181110 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:44.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.530+0000 7f12bffff700 1 -- 172.21.15.149:0/4004181110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b0005040 con 0x7f12c00f53f0 2024-02-12T16:33:44.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.530+0000 7f12bffff700 1 --2- 172.21.15.149:0/4004181110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c00f53f0 0x7f12c0109480 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f12b0005010 tx=0x7f12b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:44.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.530+0000 7f12bcff9700 1 -- 172.21.15.149:0/4004181110 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12b0009320 con 0x7f12c00f53f0 2024-02-12T16:33:44.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.530+0000 7f12c65fa700 1 -- 172.21.15.149:0/4004181110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c0060480 con 0x7f12c00f53f0 2024-02-12T16:33:44.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.530+0000 7f12bcff9700 1 -- 172.21.15.149:0/4004181110 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12b0013380 con 0x7f12c00f53f0 2024-02-12T16:33:44.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.530+0000 7f12bcff9700 1 -- 172.21.15.149:0/4004181110 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12b0014e50 con 0x7f12c00f53f0 2024-02-12T16:33:44.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.530+0000 7f12c65fa700 1 -- 172.21.15.149:0/4004181110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c0109bb0 con 0x7f12c00f53f0 2024-02-12T16:33:44.534 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.531+0000 7f12bcff9700 1 -- 172.21.15.149:0/4004181110 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12b0014420 con 0x7f12c00f53f0 2024-02-12T16:33:44.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.531+0000 7f12bcff9700 1 --2- 172.21.15.149:0/4004181110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12a802ff10 0x7f12a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:44.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.531+0000 7f12c65fa700 1 -- 172.21.15.149:0/4004181110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c0045ad0 con 0x7f12c00f53f0 2024-02-12T16:33:44.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.531+0000 7f12bcff9700 1 -- 172.21.15.149:0/4004181110 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12b0040e80 con 0x7f12c00f53f0 2024-02-12T16:33:44.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.531+0000 7f12bf7fe700 1 --2- 172.21.15.149:0/4004181110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12a802ff10 0x7f12a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:44.535 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.532+0000 7f12bf7fe700 1 --2- 172.21.15.149:0/4004181110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12a802ff10 0x7f12a80323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f12b40096a0 tx=0x7f12b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:44.538 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.536+0000 7f12bcff9700 1 -- 172.21.15.149:0/4004181110 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12b0014790 con 0x7f12c00f53f0 2024-02-12T16:33:44.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:42.312533+0000 mgr.smithi104.ohwbhf (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:44.742 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.740+0000 7f12c65fa700 1 -- 172.21.15.149:0/4004181110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12c00faf80 con 0x7f12c00f53f0 2024-02-12T16:33:44.742 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.740+0000 7f12bcff9700 1 -- 172.21.15.149:0/4004181110 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f12b001d380 con 0x7f12c00f53f0 2024-02-12T16:33:44.743 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:44.743 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:44.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.742+0000 7f12c65fa700 1 -- 172.21.15.149:0/4004181110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12a802ff10 msgr2=0x7f12a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:44.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.742+0000 7f12c65fa700 1 --2- 172.21.15.149:0/4004181110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12a802ff10 0x7f12a80323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f12b40096a0 tx=0x7f12b4006b40).stop 2024-02-12T16:33:44.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.742+0000 7f12c65fa700 1 -- 172.21.15.149:0/4004181110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c00f53f0 msgr2=0x7f12c0109480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:44.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.742+0000 7f12c65fa700 1 --2- 172.21.15.149:0/4004181110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c00f53f0 0x7f12c0109480 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f12b0005010 tx=0x7f12b000b040).stop 2024-02-12T16:33:44.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.742+0000 7f12c65fa700 1 -- 172.21.15.149:0/4004181110 shutdown_connections 2024-02-12T16:33:44.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.742+0000 7f12c65fa700 1 --2- 172.21.15.149:0/4004181110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f12a802ff10 0x7f12a80323d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:44.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.742+0000 7f12c65fa700 1 --2- 172.21.15.149:0/4004181110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f12c00f53f0 0x7f12c0109480 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:44.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.742+0000 7f12c65fa700 1 -- 172.21.15.149:0/4004181110 >> 172.21.15.149:0/4004181110 conn(0x7f12c00f0b70 msgr2=0x7f12c00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:44.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.742+0000 7f12c65fa700 1 -- 172.21.15.149:0/4004181110 shutdown_connections 2024-02-12T16:33:44.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:44.742+0000 7f12c65fa700 1 -- 172.21.15.149:0/4004181110 wait complete. 2024-02-12T16:33:44.745 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:45.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:45 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:44.741719+0000 mon.smithi104 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.149:0/4004181110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:46.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:46.389 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:46.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:44.312837+0000 mgr.smithi104.ohwbhf (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:48.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.062+0000 7fefad0cd700 1 -- 172.21.15.149:0/1295178282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa80f5410 msgr2=0x7fefa80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:48.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.062+0000 7fefad0cd700 1 --2- 172.21.15.149:0/1295178282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa80f5410 0x7fefa80f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fef98009fb0 tx=0x7fef98009b30).stop 2024-02-12T16:33:48.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.062+0000 7fefad0cd700 1 -- 172.21.15.149:0/1295178282 shutdown_connections 2024-02-12T16:33:48.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.062+0000 7fefad0cd700 1 --2- 172.21.15.149:0/1295178282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa80f5410 0x7fefa80f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:48.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.062+0000 7fefad0cd700 1 -- 172.21.15.149:0/1295178282 >> 172.21.15.149:0/1295178282 conn(0x7fefa80f0b70 msgr2=0x7fefa80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:48.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.062+0000 7fefad0cd700 1 -- 172.21.15.149:0/1295178282 shutdown_connections 2024-02-12T16:33:48.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.062+0000 7fefad0cd700 1 -- 172.21.15.149:0/1295178282 wait complete. 2024-02-12T16:33:48.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.062+0000 7fefad0cd700 1 Processor -- start 2024-02-12T16:33:48.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.062+0000 7fefad0cd700 1 -- start start 2024-02-12T16:33:48.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fefad0cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa80f5410 0x7fefa81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:48.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fefad0cd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa8105bd0 con 0x7fefa80f5410 2024-02-12T16:33:48.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fefa6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa80f5410 0x7fefa81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:48.067 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fefa6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa80f5410 0x7fefa81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:57674/0 (socket says 172.21.15.149:57674) 2024-02-12T16:33:48.067 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fefa6d9d700 1 -- 172.21.15.149:0/3252506545 learned_addr learned my addr 172.21.15.149:0/3252506545 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:48.067 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fefa6d9d700 1 -- 172.21.15.149:0/3252506545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef98005040 con 0x7fefa80f5410 2024-02-12T16:33:48.067 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fefa6d9d700 1 --2- 172.21.15.149:0/3252506545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa80f5410 0x7fefa81056e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fef98005010 tx=0x7fef9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:48.067 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fef8f7fe700 1 -- 172.21.15.149:0/3252506545 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef98009320 con 0x7fefa80f5410 2024-02-12T16:33:48.067 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fefad0cd700 1 -- 172.21.15.149:0/3252506545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa8105dd0 con 0x7fefa80f5410 2024-02-12T16:33:48.067 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fef8f7fe700 1 -- 172.21.15.149:0/3252506545 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef98013380 con 0x7fefa80f5410 2024-02-12T16:33:48.067 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fef8f7fe700 1 -- 172.21.15.149:0/3252506545 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef98014e50 con 0x7fefa80f5410 2024-02-12T16:33:48.067 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.063+0000 7fefad0cd700 1 -- 172.21.15.149:0/3252506545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa8106260 con 0x7fefa80f5410 2024-02-12T16:33:48.068 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.064+0000 7fef8f7fe700 1 -- 172.21.15.149:0/3252506545 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef980263a0 con 0x7fefa80f5410 2024-02-12T16:33:48.068 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.064+0000 7fef8f7fe700 1 --2- 172.21.15.149:0/3252506545 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fef9002ff10 0x7fef900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:48.068 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.064+0000 7fef8f7fe700 1 -- 172.21.15.149:0/3252506545 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef98041270 con 0x7fefa80f5410 2024-02-12T16:33:48.068 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.064+0000 7fefad0cd700 1 -- 172.21.15.149:0/3252506545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa8045ad0 con 0x7fefa80f5410 2024-02-12T16:33:48.068 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.065+0000 7fefa659c700 1 --2- 172.21.15.149:0/3252506545 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fef9002ff10 0x7fef900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:48.069 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.065+0000 7fefa659c700 1 --2- 172.21.15.149:0/3252506545 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fef9002ff10 0x7fef900323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fef9c0096a0 tx=0x7fef9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:48.071 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.069+0000 7fef8f7fe700 1 -- 172.21.15.149:0/3252506545 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef98012920 con 0x7fefa80f5410 2024-02-12T16:33:48.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.276+0000 7fefad0cd700 1 -- 172.21.15.149:0/3252506545 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefa8106f80 con 0x7fefa80f5410 2024-02-12T16:33:48.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.277+0000 7fef8f7fe700 1 -- 172.21.15.149:0/3252506545 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fef9802b030 con 0x7fefa80f5410 2024-02-12T16:33:48.279 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:48.279 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:48.280 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.278+0000 7fefad0cd700 1 -- 172.21.15.149:0/3252506545 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fef9002ff10 msgr2=0x7fef900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:48.280 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.278+0000 7fefad0cd700 1 --2- 172.21.15.149:0/3252506545 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fef9002ff10 0x7fef900323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fef9c0096a0 tx=0x7fef9c006b40).stop 2024-02-12T16:33:48.280 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.278+0000 7fefad0cd700 1 -- 172.21.15.149:0/3252506545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa80f5410 msgr2=0x7fefa81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:48.280 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.278+0000 7fefad0cd700 1 --2- 172.21.15.149:0/3252506545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa80f5410 0x7fefa81056e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fef98005010 tx=0x7fef9800b040).stop 2024-02-12T16:33:48.281 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.278+0000 7fefad0cd700 1 -- 172.21.15.149:0/3252506545 shutdown_connections 2024-02-12T16:33:48.281 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.278+0000 7fefad0cd700 1 --2- 172.21.15.149:0/3252506545 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fef9002ff10 0x7fef900323d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:48.281 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.278+0000 7fefad0cd700 1 --2- 172.21.15.149:0/3252506545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa80f5410 0x7fefa81056e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:48.281 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.278+0000 7fefad0cd700 1 -- 172.21.15.149:0/3252506545 >> 172.21.15.149:0/3252506545 conn(0x7fefa80f0b70 msgr2=0x7fefa80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:48.281 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.278+0000 7fefad0cd700 1 -- 172.21.15.149:0/3252506545 shutdown_connections 2024-02-12T16:33:48.281 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:48.278+0000 7fefad0cd700 1 -- 172.21.15.149:0/3252506545 wait complete. 2024-02-12T16:33:48.281 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:48.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:46.313209+0000 mgr.smithi104.ohwbhf (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:48.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:48.278009+0000 mon.smithi104 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.149:0/3252506545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:49.887 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:49.888 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:50.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:48.313543+0000 mgr.smithi104.ohwbhf (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:51.713 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.711+0000 7f61993ad700 1 -- 172.21.15.149:0/2970787633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61940f6510 msgr2=0x7f61940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:51.714 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.711+0000 7f61993ad700 1 --2- 172.21.15.149:0/2970787633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61940f6510 0x7f61940f6930 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f6184009fb0 tx=0x7f6184009b30).stop 2024-02-12T16:33:51.714 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.711+0000 7f61993ad700 1 -- 172.21.15.149:0/2970787633 shutdown_connections 2024-02-12T16:33:51.714 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.711+0000 7f61993ad700 1 --2- 172.21.15.149:0/2970787633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61940f6510 0x7f61940f6930 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:51.714 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.711+0000 7f61993ad700 1 -- 172.21.15.149:0/2970787633 >> 172.21.15.149:0/2970787633 conn(0x7f61940f1c70 msgr2=0x7f61940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:51.714 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.711+0000 7f61993ad700 1 -- 172.21.15.149:0/2970787633 shutdown_connections 2024-02-12T16:33:51.714 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.711+0000 7f61993ad700 1 -- 172.21.15.149:0/2970787633 wait complete. 2024-02-12T16:33:51.715 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.712+0000 7f61993ad700 1 Processor -- start 2024-02-12T16:33:51.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.712+0000 7f61993ad700 1 -- start start 2024-02-12T16:33:51.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.712+0000 7f61993ad700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61940f6510 0x7f61941067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:51.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.712+0000 7f61993ad700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6194106cd0 con 0x7f61940f6510 2024-02-12T16:33:51.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.713+0000 7f6192ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61940f6510 0x7f61941067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:51.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.713+0000 7f6192ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61940f6510 0x7f61941067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:57686/0 (socket says 172.21.15.149:57686) 2024-02-12T16:33:51.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.713+0000 7f6192ffd700 1 -- 172.21.15.149:0/583912178 learned_addr learned my addr 172.21.15.149:0/583912178 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:51.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.713+0000 7f6192ffd700 1 -- 172.21.15.149:0/583912178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6184005040 con 0x7f61940f6510 2024-02-12T16:33:51.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.713+0000 7f6192ffd700 1 --2- 172.21.15.149:0/583912178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61940f6510 0x7f61941067e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f6184005010 tx=0x7f618400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:51.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.713+0000 7f618b7fe700 1 -- 172.21.15.149:0/583912178 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6184009320 con 0x7f61940f6510 2024-02-12T16:33:51.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.713+0000 7f61993ad700 1 -- 172.21.15.149:0/583912178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6194106ed0 con 0x7f61940f6510 2024-02-12T16:33:51.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.713+0000 7f618b7fe700 1 -- 172.21.15.149:0/583912178 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6184013380 con 0x7f61940f6510 2024-02-12T16:33:51.717 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.713+0000 7f618b7fe700 1 -- 172.21.15.149:0/583912178 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6184014e50 con 0x7f61940f6510 2024-02-12T16:33:51.717 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.713+0000 7f61993ad700 1 -- 172.21.15.149:0/583912178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6194107360 con 0x7f61940f6510 2024-02-12T16:33:51.717 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.714+0000 7f618b7fe700 1 -- 172.21.15.149:0/583912178 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61840095f0 con 0x7f61940f6510 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.714+0000 7f61993ad700 1 -- 172.21.15.149:0/583912178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6194045ad0 con 0x7f61940f6510 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.714+0000 7f618b7fe700 1 --2- 172.21.15.149:0/583912178 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f618002ff10 0x7f61800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.714+0000 7f618b7fe700 1 -- 172.21.15.149:0/583912178 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6184040e90 con 0x7f61940f6510 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.715+0000 7f61927fc700 1 --2- 172.21.15.149:0/583912178 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f618002ff10 0x7f61800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.715+0000 7f61927fc700 1 --2- 172.21.15.149:0/583912178 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f618002ff10 0x7f61800323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f617c0096a0 tx=0x7f617c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.719+0000 7f618b7fe700 1 -- 172.21.15.149:0/583912178 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6184014420 con 0x7f61940f6510 2024-02-12T16:33:51.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.921+0000 7f61993ad700 1 -- 172.21.15.149:0/583912178 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6194108210 con 0x7f61940f6510 2024-02-12T16:33:51.923 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.921+0000 7f618b7fe700 1 -- 172.21.15.149:0/583912178 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6184010230 con 0x7f61940f6510 2024-02-12T16:33:51.923 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:51.924 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:51.925 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.923+0000 7f61993ad700 1 -- 172.21.15.149:0/583912178 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f618002ff10 msgr2=0x7f61800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:51.925 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.923+0000 7f61993ad700 1 --2- 172.21.15.149:0/583912178 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f618002ff10 0x7f61800323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f617c0096a0 tx=0x7f617c006b40).stop 2024-02-12T16:33:51.925 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.923+0000 7f61993ad700 1 -- 172.21.15.149:0/583912178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61940f6510 msgr2=0x7f61941067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:51.926 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.923+0000 7f61993ad700 1 --2- 172.21.15.149:0/583912178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61940f6510 0x7f61941067e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f6184005010 tx=0x7f618400b040).stop 2024-02-12T16:33:51.926 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.923+0000 7f61993ad700 1 -- 172.21.15.149:0/583912178 shutdown_connections 2024-02-12T16:33:51.926 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.923+0000 7f61993ad700 1 --2- 172.21.15.149:0/583912178 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f618002ff10 0x7f61800323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:51.926 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.923+0000 7f61993ad700 1 --2- 172.21.15.149:0/583912178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f61940f6510 0x7f61941067e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:51.926 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.923+0000 7f61993ad700 1 -- 172.21.15.149:0/583912178 >> 172.21.15.149:0/583912178 conn(0x7f61940f1c70 msgr2=0x7f61940f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:51.926 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.923+0000 7f61993ad700 1 -- 172.21.15.149:0/583912178 shutdown_connections 2024-02-12T16:33:51.926 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:51.923+0000 7f61993ad700 1 -- 172.21.15.149:0/583912178 wait complete. 2024-02-12T16:33:51.926 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:52.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:50.313825+0000 mgr.smithi104.ohwbhf (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:52.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:51.922567+0000 mon.smithi104 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.149:0/583912178' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:53.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:53.461 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:53.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:53 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:52.314052+0000 mgr.smithi104.ohwbhf (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:55.359 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.269+0000 7f6d95b1a700 1 -- 172.21.15.149:0/2039255607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d90060220 msgr2=0x7f6d900f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.269+0000 7f6d95b1a700 1 --2- 172.21.15.149:0/2039255607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d90060220 0x7f6d900f9150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f6d78009fb0 tx=0x7f6d78009b30).stop 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.269+0000 7f6d95b1a700 1 -- 172.21.15.149:0/2039255607 shutdown_connections 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.269+0000 7f6d95b1a700 1 --2- 172.21.15.149:0/2039255607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d90060220 0x7f6d900f9150 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.269+0000 7f6d95b1a700 1 -- 172.21.15.149:0/2039255607 >> 172.21.15.149:0/2039255607 conn(0x7f6d900f0b70 msgr2=0x7f6d900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.269+0000 7f6d95b1a700 1 -- 172.21.15.149:0/2039255607 shutdown_connections 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.269+0000 7f6d95b1a700 1 -- 172.21.15.149:0/2039255607 wait complete. 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.269+0000 7f6d95b1a700 1 Processor -- start 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.270+0000 7f6d95b1a700 1 -- start start 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.270+0000 7f6d95b1a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d90060220 0x7f6d901089b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.270+0000 7f6d95b1a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d90108ea0 con 0x7f6d90060220 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.270+0000 7f6d8f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d90060220 0x7f6d901089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:55.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.270+0000 7f6d8f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d90060220 0x7f6d901089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:52708/0 (socket says 172.21.15.149:52708) 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.270+0000 7f6d8f7fe700 1 -- 172.21.15.149:0/830622077 learned_addr learned my addr 172.21.15.149:0/830622077 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.270+0000 7f6d8f7fe700 1 -- 172.21.15.149:0/830622077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78005040 con 0x7f6d90060220 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.271+0000 7f6d8f7fe700 1 --2- 172.21.15.149:0/830622077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d90060220 0x7f6d901089b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f6d780089f0 tx=0x7f6d7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.271+0000 7f6d87fff700 1 -- 172.21.15.149:0/830622077 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d78009320 con 0x7f6d90060220 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.271+0000 7f6d87fff700 1 -- 172.21.15.149:0/830622077 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d78013380 con 0x7f6d90060220 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.271+0000 7f6d95b1a700 1 -- 172.21.15.149:0/830622077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d901090a0 con 0x7f6d90060220 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.271+0000 7f6d87fff700 1 -- 172.21.15.149:0/830622077 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d78014e20 con 0x7f6d90060220 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.271+0000 7f6d95b1a700 1 -- 172.21.15.149:0/830622077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d90109530 con 0x7f6d90060220 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.272+0000 7f6d87fff700 1 -- 172.21.15.149:0/830622077 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d780095f0 con 0x7f6d90060220 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.272+0000 7f6d87fff700 1 --2- 172.21.15.149:0/830622077 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7c02ff60 0x7f6d7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.272+0000 7f6d95b1a700 1 -- 172.21.15.149:0/830622077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d90045ad0 con 0x7f6d90060220 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.272+0000 7f6d87fff700 1 -- 172.21.15.149:0/830622077 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d78040f00 con 0x7f6d90060220 2024-02-12T16:33:55.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.272+0000 7f6d8effd700 1 --2- 172.21.15.149:0/830622077 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7c02ff60 0x7f6d7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:55.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.272+0000 7f6d8effd700 1 --2- 172.21.15.149:0/830622077 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7c02ff60 0x7f6d7c032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6d800096a0 tx=0x7f6d80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:55.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.277+0000 7f6d87fff700 1 -- 172.21.15.149:0/830622077 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d7801dd70 con 0x7f6d90060220 2024-02-12T16:33:55.483 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.481+0000 7f6d95b1a700 1 -- 172.21.15.149:0/830622077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d9010a3e0 con 0x7f6d90060220 2024-02-12T16:33:55.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.482+0000 7f6d87fff700 1 -- 172.21.15.149:0/830622077 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6d7801d980 con 0x7f6d90060220 2024-02-12T16:33:55.484 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:55.484 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:55.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.483+0000 7f6d95b1a700 1 -- 172.21.15.149:0/830622077 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7c02ff60 msgr2=0x7f6d7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:55.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.483+0000 7f6d95b1a700 1 --2- 172.21.15.149:0/830622077 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7c02ff60 0x7f6d7c032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6d800096a0 tx=0x7f6d80006b40).stop 2024-02-12T16:33:55.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.483+0000 7f6d95b1a700 1 -- 172.21.15.149:0/830622077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d90060220 msgr2=0x7f6d901089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:55.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.483+0000 7f6d95b1a700 1 --2- 172.21.15.149:0/830622077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d90060220 0x7f6d901089b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f6d780089f0 tx=0x7f6d7800b040).stop 2024-02-12T16:33:55.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.483+0000 7f6d95b1a700 1 -- 172.21.15.149:0/830622077 shutdown_connections 2024-02-12T16:33:55.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.483+0000 7f6d95b1a700 1 --2- 172.21.15.149:0/830622077 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7c02ff60 0x7f6d7c032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:55.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.483+0000 7f6d95b1a700 1 --2- 172.21.15.149:0/830622077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d90060220 0x7f6d901089b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:55.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.483+0000 7f6d95b1a700 1 -- 172.21.15.149:0/830622077 >> 172.21.15.149:0/830622077 conn(0x7f6d900f0b70 msgr2=0x7f6d900f1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:55.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.484+0000 7f6d95b1a700 1 -- 172.21.15.149:0/830622077 shutdown_connections 2024-02-12T16:33:55.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:55.484+0000 7f6d95b1a700 1 -- 172.21.15.149:0/830622077 wait complete. 2024-02-12T16:33:55.487 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:33:55.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:55 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:54.314346+0000 mgr.smithi104.ohwbhf (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:56.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:55.483228+0000 mon.smithi104 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.149:0/830622077' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:33:57.074 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:33:57.074 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:33:58.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:33:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:56.314605+0000 mgr.smithi104.ohwbhf (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:33:58.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.937+0000 7fbc1c037700 1 -- 172.21.15.149:0/1039790420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc140f5410 msgr2=0x7fbc140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:58.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.937+0000 7fbc1c037700 1 --2- 172.21.15.149:0/1039790420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc140f5410 0x7fbc140f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fbc04009fb0 tx=0x7fbc04009b30).stop 2024-02-12T16:33:58.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.937+0000 7fbc1c037700 1 -- 172.21.15.149:0/1039790420 shutdown_connections 2024-02-12T16:33:58.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.937+0000 7fbc1c037700 1 --2- 172.21.15.149:0/1039790420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc140f5410 0x7fbc140f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:58.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.937+0000 7fbc1c037700 1 -- 172.21.15.149:0/1039790420 >> 172.21.15.149:0/1039790420 conn(0x7fbc140f0b70 msgr2=0x7fbc140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:58.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.937+0000 7fbc1c037700 1 -- 172.21.15.149:0/1039790420 shutdown_connections 2024-02-12T16:33:58.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.937+0000 7fbc1c037700 1 -- 172.21.15.149:0/1039790420 wait complete. 2024-02-12T16:33:58.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.938+0000 7fbc1c037700 1 Processor -- start 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.938+0000 7fbc1c037700 1 -- start start 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.938+0000 7fbc1c037700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc140f5410 0x7fbc1410b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.938+0000 7fbc1c037700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc141072f0 con 0x7fbc140f5410 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.938+0000 7fbc19dd3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc140f5410 0x7fbc1410b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.938+0000 7fbc19dd3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc140f5410 0x7fbc1410b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:52726/0 (socket says 172.21.15.149:52726) 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.938+0000 7fbc19dd3700 1 -- 172.21.15.149:0/2908628569 learned_addr learned my addr 172.21.15.149:0/2908628569 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.939+0000 7fbc19dd3700 1 -- 172.21.15.149:0/2908628569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc04005040 con 0x7fbc140f5410 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.939+0000 7fbc19dd3700 1 --2- 172.21.15.149:0/2908628569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc140f5410 0x7fbc1410b900 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fbc04005010 tx=0x7fbc0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.939+0000 7fbc0a7fc700 1 -- 172.21.15.149:0/2908628569 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc04009320 con 0x7fbc140f5410 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.939+0000 7fbc1c037700 1 -- 172.21.15.149:0/2908628569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc14060220 con 0x7fbc140f5410 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.939+0000 7fbc0a7fc700 1 -- 172.21.15.149:0/2908628569 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc04013380 con 0x7fbc140f5410 2024-02-12T16:33:58.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.939+0000 7fbc0a7fc700 1 -- 172.21.15.149:0/2908628569 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc04014e50 con 0x7fbc140f5410 2024-02-12T16:33:58.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.939+0000 7fbc1c037700 1 -- 172.21.15.149:0/2908628569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1410bdf0 con 0x7fbc140f5410 2024-02-12T16:33:58.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.940+0000 7fbc0a7fc700 1 -- 172.21.15.149:0/2908628569 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc040095f0 con 0x7fbc140f5410 2024-02-12T16:33:58.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.940+0000 7fbc1c037700 1 -- 172.21.15.149:0/2908628569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc14045ad0 con 0x7fbc140f5410 2024-02-12T16:33:58.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.940+0000 7fbc0a7fc700 1 --2- 172.21.15.149:0/2908628569 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbc0002ff10 0x7fbc000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:33:58.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.940+0000 7fbc0a7fc700 1 -- 172.21.15.149:0/2908628569 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc04040f60 con 0x7fbc140f5410 2024-02-12T16:33:58.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.940+0000 7fbc195d2700 1 --2- 172.21.15.149:0/2908628569 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbc0002ff10 0x7fbc000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:33:58.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.940+0000 7fbc195d2700 1 --2- 172.21.15.149:0/2908628569 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbc0002ff10 0x7fbc000323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbc100096a0 tx=0x7fbc10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:33:58.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:58.944+0000 7fbc0a7fc700 1 -- 172.21.15.149:0/2908628569 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc04014420 con 0x7fbc140f5410 2024-02-12T16:33:59.150 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.148+0000 7fbc1c037700 1 -- 172.21.15.149:0/2908628569 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc1410c5b0 con 0x7fbc140f5410 2024-02-12T16:33:59.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.149+0000 7fbc0a7fc700 1 -- 172.21.15.149:0/2908628569 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbc04014420 con 0x7fbc140f5410 2024-02-12T16:33:59.151 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:33:59.151 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:33:59.153 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.151+0000 7fbc1c037700 1 -- 172.21.15.149:0/2908628569 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbc0002ff10 msgr2=0x7fbc000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:59.153 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.151+0000 7fbc1c037700 1 --2- 172.21.15.149:0/2908628569 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbc0002ff10 0x7fbc000323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbc100096a0 tx=0x7fbc10006b40).stop 2024-02-12T16:33:59.153 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.151+0000 7fbc1c037700 1 -- 172.21.15.149:0/2908628569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc140f5410 msgr2=0x7fbc1410b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:33:59.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.151+0000 7fbc1c037700 1 --2- 172.21.15.149:0/2908628569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc140f5410 0x7fbc1410b900 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fbc04005010 tx=0x7fbc0400b040).stop 2024-02-12T16:33:59.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.151+0000 7fbc1c037700 1 -- 172.21.15.149:0/2908628569 shutdown_connections 2024-02-12T16:33:59.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.151+0000 7fbc1c037700 1 --2- 172.21.15.149:0/2908628569 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbc0002ff10 0x7fbc000323d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:59.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.151+0000 7fbc1c037700 1 --2- 172.21.15.149:0/2908628569 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbc140f5410 0x7fbc1410b900 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:33:59.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.151+0000 7fbc1c037700 1 -- 172.21.15.149:0/2908628569 >> 172.21.15.149:0/2908628569 conn(0x7fbc140f0b70 msgr2=0x7fbc140f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:33:59.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.151+0000 7fbc1c037700 1 -- 172.21.15.149:0/2908628569 shutdown_connections 2024-02-12T16:33:59.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:33:59.151+0000 7fbc1c037700 1 -- 172.21.15.149:0/2908628569 wait complete. 2024-02-12T16:33:59.154 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:00.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:33:58.314828+0000 mgr.smithi104.ohwbhf (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:00.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:33:59.150039+0000 mon.smithi104 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.149:0/2908628569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:00.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:00.694 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:02.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.405+0000 7f4aabfd1700 1 -- 172.21.15.149:0/3394071017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4aa40f5410 msgr2=0x7f4aa40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:02.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.405+0000 7f4aabfd1700 1 --2- 172.21.15.149:0/3394071017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4aa40f5410 0x7f4aa40f5830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f4a94009fb0 tx=0x7f4a94009b30).stop 2024-02-12T16:34:02.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.405+0000 7f4aabfd1700 1 -- 172.21.15.149:0/3394071017 shutdown_connections 2024-02-12T16:34:02.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.405+0000 7f4aabfd1700 1 --2- 172.21.15.149:0/3394071017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4aa40f5410 0x7f4aa40f5830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:02.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.405+0000 7f4aabfd1700 1 -- 172.21.15.149:0/3394071017 >> 172.21.15.149:0/3394071017 conn(0x7f4aa40f0b70 msgr2=0x7f4aa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:02.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.405+0000 7f4aabfd1700 1 -- 172.21.15.149:0/3394071017 shutdown_connections 2024-02-12T16:34:02.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.405+0000 7f4aabfd1700 1 -- 172.21.15.149:0/3394071017 wait complete. 2024-02-12T16:34:02.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.406+0000 7f4aabfd1700 1 Processor -- start 2024-02-12T16:34:02.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.406+0000 7f4aabfd1700 1 -- start start 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.406+0000 7f4aabfd1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4aa40f5410 0x7f4aa41067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.406+0000 7f4aabfd1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa4106ca0 con 0x7f4aa40f5410 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.406+0000 7f4aa9d6d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4aa40f5410 0x7f4aa41067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.406+0000 7f4aa9d6d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4aa40f5410 0x7f4aa41067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:52742/0 (socket says 172.21.15.149:52742) 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.406+0000 7f4aa9d6d700 1 -- 172.21.15.149:0/2713182263 learned_addr learned my addr 172.21.15.149:0/2713182263 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.406+0000 7f4aa9d6d700 1 -- 172.21.15.149:0/2713182263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a94005040 con 0x7f4aa40f5410 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.407+0000 7f4aa9d6d700 1 --2- 172.21.15.149:0/2713182263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4aa40f5410 0x7f4aa41067b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f4a94005010 tx=0x7f4a9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.407+0000 7f4a9a7fc700 1 -- 172.21.15.149:0/2713182263 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a94009320 con 0x7f4aa40f5410 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.407+0000 7f4aabfd1700 1 -- 172.21.15.149:0/2713182263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa4106ea0 con 0x7f4aa40f5410 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.407+0000 7f4a9a7fc700 1 -- 172.21.15.149:0/2713182263 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a94013380 con 0x7f4aa40f5410 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.407+0000 7f4aabfd1700 1 -- 172.21.15.149:0/2713182263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa4107330 con 0x7f4aa40f5410 2024-02-12T16:34:02.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.407+0000 7f4a9a7fc700 1 -- 172.21.15.149:0/2713182263 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a94014e50 con 0x7f4aa40f5410 2024-02-12T16:34:02.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.408+0000 7f4a9a7fc700 1 -- 172.21.15.149:0/2713182263 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a94014420 con 0x7f4aa40f5410 2024-02-12T16:34:02.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.408+0000 7f4a87fff700 1 -- 172.21.15.149:0/2713182263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa4045ad0 con 0x7f4aa40f5410 2024-02-12T16:34:02.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.408+0000 7f4a9a7fc700 1 --2- 172.21.15.149:0/2713182263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a9002ff10 0x7f4a900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:02.412 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.408+0000 7f4a9a7fc700 1 -- 172.21.15.149:0/2713182263 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a94040ff0 con 0x7f4aa40f5410 2024-02-12T16:34:02.412 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.408+0000 7f4aa956c700 1 --2- 172.21.15.149:0/2713182263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a9002ff10 0x7f4a900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:02.412 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.408+0000 7f4aa956c700 1 --2- 172.21.15.149:0/2713182263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a9002ff10 0x7f4a900323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f4aa00096a0 tx=0x7f4aa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:02.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.413+0000 7f4a9a7fc700 1 -- 172.21.15.149:0/2713182263 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a940095f0 con 0x7f4aa40f5410 2024-02-12T16:34:02.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:00.315137+0000 mgr.smithi104.ohwbhf (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:02.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.616+0000 7f4a87fff700 1 -- 172.21.15.149:0/2713182263 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4aa40fafa0 con 0x7f4aa40f5410 2024-02-12T16:34:02.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.616+0000 7f4a9a7fc700 1 -- 172.21.15.149:0/2713182263 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4a9401db70 con 0x7f4aa40f5410 2024-02-12T16:34:02.618 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:02.619 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:02.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.618+0000 7f4a87fff700 1 -- 172.21.15.149:0/2713182263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a9002ff10 msgr2=0x7f4a900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:02.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.618+0000 7f4a87fff700 1 --2- 172.21.15.149:0/2713182263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a9002ff10 0x7f4a900323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f4aa00096a0 tx=0x7f4aa0006b40).stop 2024-02-12T16:34:02.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.618+0000 7f4a87fff700 1 -- 172.21.15.149:0/2713182263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4aa40f5410 msgr2=0x7f4aa41067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:02.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.618+0000 7f4a87fff700 1 --2- 172.21.15.149:0/2713182263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4aa40f5410 0x7f4aa41067b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f4a94005010 tx=0x7f4a9400b040).stop 2024-02-12T16:34:02.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.618+0000 7f4a87fff700 1 -- 172.21.15.149:0/2713182263 shutdown_connections 2024-02-12T16:34:02.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.618+0000 7f4a87fff700 1 --2- 172.21.15.149:0/2713182263 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f4a9002ff10 0x7f4a900323d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:02.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.618+0000 7f4a87fff700 1 --2- 172.21.15.149:0/2713182263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4aa40f5410 0x7f4aa41067b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:02.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.618+0000 7f4a87fff700 1 -- 172.21.15.149:0/2713182263 >> 172.21.15.149:0/2713182263 conn(0x7f4aa40f0b70 msgr2=0x7f4aa405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:02.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.618+0000 7f4a87fff700 1 -- 172.21.15.149:0/2713182263 shutdown_connections 2024-02-12T16:34:02.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:02.618+0000 7f4a87fff700 1 -- 172.21.15.149:0/2713182263 wait complete. 2024-02-12T16:34:02.621 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:03.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:03 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:02.617609+0000 mon.smithi104 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.149:0/2713182263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:04.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:04.288 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:04.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:02.315378+0000 mgr.smithi104.ohwbhf (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:06.059 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.055+0000 7f7fe1279700 1 -- 172.21.15.149:0/1045159955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fdc060280 msgr2=0x7f7fdc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:06.059 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.055+0000 7f7fe1279700 1 --2- 172.21.15.149:0/1045159955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fdc060280 0x7f7fdc0f90f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f7fd0009fb0 tx=0x7f7fd0009b30).stop 2024-02-12T16:34:06.059 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.055+0000 7f7fe1279700 1 -- 172.21.15.149:0/1045159955 shutdown_connections 2024-02-12T16:34:06.059 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.055+0000 7f7fe1279700 1 --2- 172.21.15.149:0/1045159955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fdc060280 0x7f7fdc0f90f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:06.059 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.055+0000 7f7fe1279700 1 -- 172.21.15.149:0/1045159955 >> 172.21.15.149:0/1045159955 conn(0x7f7fdc0f0b70 msgr2=0x7f7fdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:06.059 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.055+0000 7f7fe1279700 1 -- 172.21.15.149:0/1045159955 shutdown_connections 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.055+0000 7f7fe1279700 1 -- 172.21.15.149:0/1045159955 wait complete. 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.056+0000 7f7fe1279700 1 Processor -- start 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.056+0000 7f7fe1279700 1 -- start start 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.056+0000 7f7fe1279700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fdc060280 0x7f7fdc106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.056+0000 7f7fe1279700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc106c70 con 0x7f7fdc060280 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.056+0000 7f7fdaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fdc060280 0x7f7fdc106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.056+0000 7f7fdaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fdc060280 0x7f7fdc106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:44798/0 (socket says 172.21.15.149:44798) 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.056+0000 7f7fdaffd700 1 -- 172.21.15.149:0/545212114 learned_addr learned my addr 172.21.15.149:0/545212114 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.057+0000 7f7fdaffd700 1 -- 172.21.15.149:0/545212114 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd0005040 con 0x7f7fdc060280 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.057+0000 7f7fdaffd700 1 --2- 172.21.15.149:0/545212114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fdc060280 0x7f7fdc106780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f7fd0009830 tx=0x7f7fd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.057+0000 7f7fc37fe700 1 -- 172.21.15.149:0/545212114 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fd0009320 con 0x7f7fdc060280 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.057+0000 7f7fe1279700 1 -- 172.21.15.149:0/545212114 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fdc106e70 con 0x7f7fdc060280 2024-02-12T16:34:06.060 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.057+0000 7f7fc37fe700 1 -- 172.21.15.149:0/545212114 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fd0013380 con 0x7f7fdc060280 2024-02-12T16:34:06.061 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.057+0000 7f7fc37fe700 1 -- 172.21.15.149:0/545212114 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fd0014420 con 0x7f7fdc060280 2024-02-12T16:34:06.061 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.057+0000 7f7fe1279700 1 -- 172.21.15.149:0/545212114 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fdc107300 con 0x7f7fdc060280 2024-02-12T16:34:06.061 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.058+0000 7f7fc37fe700 1 -- 172.21.15.149:0/545212114 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fd0014580 con 0x7f7fdc060280 2024-02-12T16:34:06.062 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.058+0000 7f7fe1279700 1 -- 172.21.15.149:0/545212114 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc045ad0 con 0x7f7fdc060280 2024-02-12T16:34:06.062 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.058+0000 7f7fc37fe700 1 --2- 172.21.15.149:0/545212114 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7fc402ff60 0x7f7fc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:06.062 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.058+0000 7f7fc37fe700 1 -- 172.21.15.149:0/545212114 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fd0040ed0 con 0x7f7fdc060280 2024-02-12T16:34:06.062 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.058+0000 7f7fda7fc700 1 --2- 172.21.15.149:0/545212114 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7fc402ff60 0x7f7fc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:06.062 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.059+0000 7f7fda7fc700 1 --2- 172.21.15.149:0/545212114 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7fc402ff60 0x7f7fc4032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f7fcc0096a0 tx=0x7f7fcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:06.064 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.063+0000 7f7fc37fe700 1 -- 172.21.15.149:0/545212114 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fd001c890 con 0x7f7fdc060280 2024-02-12T16:34:06.268 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.266+0000 7f7fe1279700 1 -- 172.21.15.149:0/545212114 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fdc0fc090 con 0x7f7fdc060280 2024-02-12T16:34:06.269 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.267+0000 7f7fc37fe700 1 -- 172.21.15.149:0/545212114 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7fd002c2d0 con 0x7f7fdc060280 2024-02-12T16:34:06.269 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:06.269 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:06.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.268+0000 7f7fe1279700 1 -- 172.21.15.149:0/545212114 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7fc402ff60 msgr2=0x7f7fc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:06.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.268+0000 7f7fe1279700 1 --2- 172.21.15.149:0/545212114 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7fc402ff60 0x7f7fc4032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f7fcc0096a0 tx=0x7f7fcc006b40).stop 2024-02-12T16:34:06.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.268+0000 7f7fe1279700 1 -- 172.21.15.149:0/545212114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fdc060280 msgr2=0x7f7fdc106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:06.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.268+0000 7f7fe1279700 1 --2- 172.21.15.149:0/545212114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fdc060280 0x7f7fdc106780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f7fd0009830 tx=0x7f7fd000b040).stop 2024-02-12T16:34:06.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.268+0000 7f7fe1279700 1 -- 172.21.15.149:0/545212114 shutdown_connections 2024-02-12T16:34:06.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.268+0000 7f7fe1279700 1 --2- 172.21.15.149:0/545212114 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f7fc402ff60 0x7f7fc4032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:06.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.268+0000 7f7fe1279700 1 --2- 172.21.15.149:0/545212114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7fdc060280 0x7f7fdc106780 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:06.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.268+0000 7f7fe1279700 1 -- 172.21.15.149:0/545212114 >> 172.21.15.149:0/545212114 conn(0x7f7fdc0f0b70 msgr2=0x7f7fdc0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:06.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.268+0000 7f7fe1279700 1 -- 172.21.15.149:0/545212114 shutdown_connections 2024-02-12T16:34:06.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:06.268+0000 7f7fe1279700 1 -- 172.21.15.149:0/545212114 wait complete. 2024-02-12T16:34:06.272 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:06.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:04.315652+0000 mgr.smithi104.ohwbhf (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:07.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:07 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:06.267936+0000 mon.smithi104 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.149:0/545212114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:07.971 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:07.972 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:08.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:06.316007+0000 mgr.smithi104.ohwbhf (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:09.828 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.825+0000 7f2328845700 1 -- 172.21.15.149:0/1883030983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240f51a0 msgr2=0x7f23240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:09.828 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.825+0000 7f2328845700 1 --2- 172.21.15.149:0/1883030983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240f51a0 0x7f23240f55c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f230c009fb0 tx=0x7f230c009b30).stop 2024-02-12T16:34:09.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.825+0000 7f2328845700 1 -- 172.21.15.149:0/1883030983 shutdown_connections 2024-02-12T16:34:09.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.825+0000 7f2328845700 1 --2- 172.21.15.149:0/1883030983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240f51a0 0x7f23240f55c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:09.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.825+0000 7f2328845700 1 -- 172.21.15.149:0/1883030983 >> 172.21.15.149:0/1883030983 conn(0x7f23240f09a0 msgr2=0x7f23240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:09.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.826+0000 7f2328845700 1 -- 172.21.15.149:0/1883030983 shutdown_connections 2024-02-12T16:34:09.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.826+0000 7f2328845700 1 -- 172.21.15.149:0/1883030983 wait complete. 2024-02-12T16:34:09.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.826+0000 7f2328845700 1 Processor -- start 2024-02-12T16:34:09.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.826+0000 7f2328845700 1 -- start start 2024-02-12T16:34:09.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.826+0000 7f2328845700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240f51a0 0x7f2324109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:09.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.826+0000 7f2328845700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324060280 con 0x7f23240f51a0 2024-02-12T16:34:09.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.827+0000 7f2322d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240f51a0 0x7f2324109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:09.829 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.827+0000 7f2322d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240f51a0 0x7f2324109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:44828/0 (socket says 172.21.15.149:44828) 2024-02-12T16:34:09.830 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.827+0000 7f2322d9d700 1 -- 172.21.15.149:0/1744824534 learned_addr learned my addr 172.21.15.149:0/1744824534 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:09.831 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.827+0000 7f2322d9d700 1 -- 172.21.15.149:0/1744824534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c005040 con 0x7f23240f51a0 2024-02-12T16:34:09.831 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.827+0000 7f2322d9d700 1 --2- 172.21.15.149:0/1744824534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240f51a0 0x7f2324109290 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f230c0089f0 tx=0x7f230c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:09.831 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.827+0000 7f231b7fe700 1 -- 172.21.15.149:0/1744824534 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f230c009320 con 0x7f23240f51a0 2024-02-12T16:34:09.831 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.827+0000 7f2328845700 1 -- 172.21.15.149:0/1744824534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2324060480 con 0x7f23240f51a0 2024-02-12T16:34:09.831 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.827+0000 7f231b7fe700 1 -- 172.21.15.149:0/1744824534 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f230c013380 con 0x7f23240f51a0 2024-02-12T16:34:09.831 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.827+0000 7f231b7fe700 1 -- 172.21.15.149:0/1744824534 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f230c014e20 con 0x7f23240f51a0 2024-02-12T16:34:09.832 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.827+0000 7f2328845700 1 -- 172.21.15.149:0/1744824534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23241099c0 con 0x7f23240f51a0 2024-02-12T16:34:09.832 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.828+0000 7f231b7fe700 1 -- 172.21.15.149:0/1744824534 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f230c0095f0 con 0x7f23240f51a0 2024-02-12T16:34:09.832 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.828+0000 7f2328845700 1 -- 172.21.15.149:0/1744824534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2324045ad0 con 0x7f23240f51a0 2024-02-12T16:34:09.832 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.828+0000 7f231b7fe700 1 --2- 172.21.15.149:0/1744824534 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f231002ff10 0x7f23100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:09.832 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.828+0000 7f231b7fe700 1 -- 172.21.15.149:0/1744824534 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f230c040f50 con 0x7f23240f51a0 2024-02-12T16:34:09.832 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.828+0000 7f232259c700 1 --2- 172.21.15.149:0/1744824534 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f231002ff10 0x7f23100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:09.832 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.829+0000 7f232259c700 1 --2- 172.21.15.149:0/1744824534 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f231002ff10 0x7f23100323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f23140096a0 tx=0x7f2314006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:09.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:09.833+0000 7f231b7fe700 1 -- 172.21.15.149:0/1744824534 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f230c014420 con 0x7f23240f51a0 2024-02-12T16:34:10.037 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.035+0000 7f2328845700 1 -- 172.21.15.149:0/1744824534 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23240fad30 con 0x7f23240f51a0 2024-02-12T16:34:10.037 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.035+0000 7f231b7fe700 1 -- 172.21.15.149:0/1744824534 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f230c01d980 con 0x7f23240f51a0 2024-02-12T16:34:10.038 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:10.038 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:10.039 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.037+0000 7f2328845700 1 -- 172.21.15.149:0/1744824534 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f231002ff10 msgr2=0x7f23100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:10.039 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.037+0000 7f2328845700 1 --2- 172.21.15.149:0/1744824534 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f231002ff10 0x7f23100323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f23140096a0 tx=0x7f2314006b40).stop 2024-02-12T16:34:10.039 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.037+0000 7f2328845700 1 -- 172.21.15.149:0/1744824534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240f51a0 msgr2=0x7f2324109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:10.039 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.037+0000 7f2328845700 1 --2- 172.21.15.149:0/1744824534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240f51a0 0x7f2324109290 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f230c0089f0 tx=0x7f230c00b040).stop 2024-02-12T16:34:10.039 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.037+0000 7f2328845700 1 -- 172.21.15.149:0/1744824534 shutdown_connections 2024-02-12T16:34:10.040 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.037+0000 7f2328845700 1 --2- 172.21.15.149:0/1744824534 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f231002ff10 0x7f23100323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:10.040 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.037+0000 7f2328845700 1 --2- 172.21.15.149:0/1744824534 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23240f51a0 0x7f2324109290 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:10.040 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.037+0000 7f2328845700 1 -- 172.21.15.149:0/1744824534 >> 172.21.15.149:0/1744824534 conn(0x7f23240f09a0 msgr2=0x7f23240f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:10.040 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.037+0000 7f2328845700 1 -- 172.21.15.149:0/1744824534 shutdown_connections 2024-02-12T16:34:10.040 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:10.037+0000 7f2328845700 1 -- 172.21.15.149:0/1744824534 wait complete. 2024-02-12T16:34:10.040 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:10.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:08.316292+0000 mgr.smithi104.ohwbhf (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:10.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:10.036654+0000 mon.smithi104 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.149:0/1744824534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:11.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:11.714 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:12.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:10.316521+0000 mgr.smithi104.ohwbhf (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:13.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.507+0000 7f8df9851700 1 -- 172.21.15.149:0/3751270192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df40f5410 msgr2=0x7f8df40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:13.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.507+0000 7f8df9851700 1 --2- 172.21.15.149:0/3751270192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df40f5410 0x7f8df40f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f8de4009fb0 tx=0x7f8de4009b30).stop 2024-02-12T16:34:13.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.507+0000 7f8df9851700 1 -- 172.21.15.149:0/3751270192 shutdown_connections 2024-02-12T16:34:13.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.507+0000 7f8df9851700 1 --2- 172.21.15.149:0/3751270192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df40f5410 0x7f8df40f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:13.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.507+0000 7f8df9851700 1 -- 172.21.15.149:0/3751270192 >> 172.21.15.149:0/3751270192 conn(0x7f8df40f0b70 msgr2=0x7f8df40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:13.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.508+0000 7f8df9851700 1 -- 172.21.15.149:0/3751270192 shutdown_connections 2024-02-12T16:34:13.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.508+0000 7f8df9851700 1 -- 172.21.15.149:0/3751270192 wait complete. 2024-02-12T16:34:13.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.508+0000 7f8df9851700 1 Processor -- start 2024-02-12T16:34:13.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.508+0000 7f8df9851700 1 -- start start 2024-02-12T16:34:13.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.509+0000 7f8df9851700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df40f5410 0x7f8df4109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:13.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.509+0000 7f8df9851700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df4060240 con 0x7f8df40f5410 2024-02-12T16:34:13.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.509+0000 7f8df2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df40f5410 0x7f8df4109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:13.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.509+0000 7f8df2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df40f5410 0x7f8df4109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:44844/0 (socket says 172.21.15.149:44844) 2024-02-12T16:34:13.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.509+0000 7f8df2ffd700 1 -- 172.21.15.149:0/3817252915 learned_addr learned my addr 172.21.15.149:0/3817252915 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:13.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.509+0000 7f8df2ffd700 1 -- 172.21.15.149:0/3817252915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de4005040 con 0x7f8df40f5410 2024-02-12T16:34:13.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.509+0000 7f8df2ffd700 1 --2- 172.21.15.149:0/3817252915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df40f5410 0x7f8df4109520 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f8de4005010 tx=0x7f8de400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:13.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.509+0000 7f8deb7fe700 1 -- 172.21.15.149:0/3817252915 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8de4009320 con 0x7f8df40f5410 2024-02-12T16:34:13.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.509+0000 7f8df9851700 1 -- 172.21.15.149:0/3817252915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df4060440 con 0x7f8df40f5410 2024-02-12T16:34:13.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.510+0000 7f8deb7fe700 1 -- 172.21.15.149:0/3817252915 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8de4013380 con 0x7f8df40f5410 2024-02-12T16:34:13.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.510+0000 7f8deb7fe700 1 -- 172.21.15.149:0/3817252915 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8de4014e50 con 0x7f8df40f5410 2024-02-12T16:34:13.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.510+0000 7f8df9851700 1 -- 172.21.15.149:0/3817252915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df4109bd0 con 0x7f8df40f5410 2024-02-12T16:34:13.514 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.510+0000 7f8deb7fe700 1 -- 172.21.15.149:0/3817252915 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8de40095f0 con 0x7f8df40f5410 2024-02-12T16:34:13.514 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.510+0000 7f8df9851700 1 -- 172.21.15.149:0/3817252915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df4045ad0 con 0x7f8df40f5410 2024-02-12T16:34:13.514 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.510+0000 7f8deb7fe700 1 --2- 172.21.15.149:0/3817252915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8de002ff10 0x7f8de00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:13.514 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.511+0000 7f8deb7fe700 1 -- 172.21.15.149:0/3817252915 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8de4040fa0 con 0x7f8df40f5410 2024-02-12T16:34:13.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.511+0000 7f8df27fc700 1 --2- 172.21.15.149:0/3817252915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8de002ff10 0x7f8de00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:13.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.511+0000 7f8df27fc700 1 --2- 172.21.15.149:0/3817252915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8de002ff10 0x7f8de00323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f8ddc0096a0 tx=0x7f8ddc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:13.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.515+0000 7f8deb7fe700 1 -- 172.21.15.149:0/3817252915 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8de4014420 con 0x7f8df40f5410 2024-02-12T16:34:13.725 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.723+0000 7f8df9851700 1 -- 172.21.15.149:0/3817252915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8df410a3f0 con 0x7f8df40f5410 2024-02-12T16:34:13.736 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.724+0000 7f8deb7fe700 1 -- 172.21.15.149:0/3817252915 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8de4010230 con 0x7f8df40f5410 2024-02-12T16:34:13.736 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:13.737 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:13.738 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.725+0000 7f8df9851700 1 -- 172.21.15.149:0/3817252915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8de002ff10 msgr2=0x7f8de00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:13.738 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.725+0000 7f8df9851700 1 --2- 172.21.15.149:0/3817252915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8de002ff10 0x7f8de00323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f8ddc0096a0 tx=0x7f8ddc006b40).stop 2024-02-12T16:34:13.738 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.725+0000 7f8df9851700 1 -- 172.21.15.149:0/3817252915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df40f5410 msgr2=0x7f8df4109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:13.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.725+0000 7f8df9851700 1 --2- 172.21.15.149:0/3817252915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df40f5410 0x7f8df4109520 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f8de4005010 tx=0x7f8de400b040).stop 2024-02-12T16:34:13.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.725+0000 7f8df9851700 1 -- 172.21.15.149:0/3817252915 shutdown_connections 2024-02-12T16:34:13.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.725+0000 7f8df9851700 1 --2- 172.21.15.149:0/3817252915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8de002ff10 0x7f8de00323d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:13.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.725+0000 7f8df9851700 1 --2- 172.21.15.149:0/3817252915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8df40f5410 0x7f8df4109520 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:13.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.725+0000 7f8df9851700 1 -- 172.21.15.149:0/3817252915 >> 172.21.15.149:0/3817252915 conn(0x7f8df40f0b70 msgr2=0x7f8df40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:13.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.725+0000 7f8df9851700 1 -- 172.21.15.149:0/3817252915 shutdown_connections 2024-02-12T16:34:13.740 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:13.725+0000 7f8df9851700 1 -- 172.21.15.149:0/3817252915 wait complete. 2024-02-12T16:34:13.740 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:14.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:12.316803+0000 mgr.smithi104.ohwbhf (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:14.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:13.724936+0000 mon.smithi104 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.149:0/3817252915' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:15.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:15.380 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:16.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:14.317128+0000 mgr.smithi104.ohwbhf (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:17.225 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.223+0000 7f5f1d63c700 1 -- 172.21.15.149:0/386891437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f180f6510 msgr2=0x7f5f180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:17.225 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.223+0000 7f5f1d63c700 1 --2- 172.21.15.149:0/386891437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f180f6510 0x7f5f180f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f5f0c009fb0 tx=0x7f5f0c009b30).stop 2024-02-12T16:34:17.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.223+0000 7f5f1d63c700 1 -- 172.21.15.149:0/386891437 shutdown_connections 2024-02-12T16:34:17.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.223+0000 7f5f1d63c700 1 --2- 172.21.15.149:0/386891437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f180f6510 0x7f5f180f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:17.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.223+0000 7f5f1d63c700 1 -- 172.21.15.149:0/386891437 >> 172.21.15.149:0/386891437 conn(0x7f5f180f1c70 msgr2=0x7f5f180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:17.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.223+0000 7f5f1d63c700 1 -- 172.21.15.149:0/386891437 shutdown_connections 2024-02-12T16:34:17.226 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.223+0000 7f5f1d63c700 1 -- 172.21.15.149:0/386891437 wait complete. 2024-02-12T16:34:17.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.224+0000 7f5f1d63c700 1 Processor -- start 2024-02-12T16:34:17.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.224+0000 7f5f1d63c700 1 -- start start 2024-02-12T16:34:17.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.224+0000 7f5f1d63c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f180f6510 0x7f5f181089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:17.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.224+0000 7f5f1d63c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f18108ed0 con 0x7f5f180f6510 2024-02-12T16:34:17.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.224+0000 7f5f16ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f180f6510 0x7f5f181089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:17.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.224+0000 7f5f16ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f180f6510 0x7f5f181089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:52144/0 (socket says 172.21.15.149:52144) 2024-02-12T16:34:17.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.224+0000 7f5f16ffd700 1 -- 172.21.15.149:0/3526315110 learned_addr learned my addr 172.21.15.149:0/3526315110 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.225+0000 7f5f16ffd700 1 -- 172.21.15.149:0/3526315110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c005040 con 0x7f5f180f6510 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.225+0000 7f5f16ffd700 1 --2- 172.21.15.149:0/3526315110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f180f6510 0x7f5f181089e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f5f0c005010 tx=0x7f5f0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.225+0000 7f5eff7fe700 1 -- 172.21.15.149:0/3526315110 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f0c009320 con 0x7f5f180f6510 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.225+0000 7f5f1d63c700 1 -- 172.21.15.149:0/3526315110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f181090d0 con 0x7f5f180f6510 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.225+0000 7f5eff7fe700 1 -- 172.21.15.149:0/3526315110 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f0c013380 con 0x7f5f180f6510 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.225+0000 7f5eff7fe700 1 -- 172.21.15.149:0/3526315110 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f0c014e50 con 0x7f5f180f6510 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.225+0000 7f5f1d63c700 1 -- 172.21.15.149:0/3526315110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f18109560 con 0x7f5f180f6510 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.226+0000 7f5eff7fe700 1 -- 172.21.15.149:0/3526315110 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f0c014420 con 0x7f5f180f6510 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.226+0000 7f5f1d63c700 1 -- 172.21.15.149:0/3526315110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f18045ad0 con 0x7f5f180f6510 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.226+0000 7f5eff7fe700 1 --2- 172.21.15.149:0/3526315110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5f0002ff10 0x7f5f000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.226+0000 7f5eff7fe700 1 -- 172.21.15.149:0/3526315110 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f0c040f70 con 0x7f5f180f6510 2024-02-12T16:34:17.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.226+0000 7f5f167fc700 1 --2- 172.21.15.149:0/3526315110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5f0002ff10 0x7f5f000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:17.230 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.227+0000 7f5f167fc700 1 --2- 172.21.15.149:0/3526315110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5f0002ff10 0x7f5f000323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f5f080096a0 tx=0x7f5f08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:17.232 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.231+0000 7f5eff7fe700 1 -- 172.21.15.149:0/3526315110 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f0c0095f0 con 0x7f5f180f6510 2024-02-12T16:34:17.439 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.437+0000 7f5f1d63c700 1 -- 172.21.15.149:0/3526315110 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f1810a410 con 0x7f5f180f6510 2024-02-12T16:34:17.439 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.438+0000 7f5eff7fe700 1 -- 172.21.15.149:0/3526315110 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5f0c01db70 con 0x7f5f180f6510 2024-02-12T16:34:17.440 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:17.440 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:17.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.439+0000 7f5f1d63c700 1 -- 172.21.15.149:0/3526315110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5f0002ff10 msgr2=0x7f5f000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:17.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.439+0000 7f5f1d63c700 1 --2- 172.21.15.149:0/3526315110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5f0002ff10 0x7f5f000323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f5f080096a0 tx=0x7f5f08006b40).stop 2024-02-12T16:34:17.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.439+0000 7f5f1d63c700 1 -- 172.21.15.149:0/3526315110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f180f6510 msgr2=0x7f5f181089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:17.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.439+0000 7f5f1d63c700 1 --2- 172.21.15.149:0/3526315110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f180f6510 0x7f5f181089e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f5f0c005010 tx=0x7f5f0c00b040).stop 2024-02-12T16:34:17.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.439+0000 7f5f1d63c700 1 -- 172.21.15.149:0/3526315110 shutdown_connections 2024-02-12T16:34:17.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.439+0000 7f5f1d63c700 1 --2- 172.21.15.149:0/3526315110 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5f0002ff10 0x7f5f000323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:17.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.439+0000 7f5f1d63c700 1 --2- 172.21.15.149:0/3526315110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f180f6510 0x7f5f181089e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:17.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.439+0000 7f5f1d63c700 1 -- 172.21.15.149:0/3526315110 >> 172.21.15.149:0/3526315110 conn(0x7f5f180f1c70 msgr2=0x7f5f180f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:17.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.439+0000 7f5f1d63c700 1 -- 172.21.15.149:0/3526315110 shutdown_connections 2024-02-12T16:34:17.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:17.439+0000 7f5f1d63c700 1 -- 172.21.15.149:0/3526315110 wait complete. 2024-02-12T16:34:17.444 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:18.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:16.317368+0000 mgr.smithi104.ohwbhf (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:18.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:17.438910+0000 mon.smithi104 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.149:0/3526315110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:19.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:19.032 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:20.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:18.317687+0000 mgr.smithi104.ohwbhf (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:20.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.847+0000 7f1f490cc700 1 -- 172.21.15.149:0/1881961795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f440f51a0 msgr2=0x7f1f440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:20.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.847+0000 7f1f490cc700 1 --2- 172.21.15.149:0/1881961795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f440f51a0 0x7f1f440f55c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f1f2c009fb0 tx=0x7f1f2c009b30).stop 2024-02-12T16:34:20.848 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.847+0000 7f1f490cc700 1 -- 172.21.15.149:0/1881961795 shutdown_connections 2024-02-12T16:34:20.849 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.847+0000 7f1f490cc700 1 --2- 172.21.15.149:0/1881961795 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f440f51a0 0x7f1f440f55c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:20.849 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.847+0000 7f1f490cc700 1 -- 172.21.15.149:0/1881961795 >> 172.21.15.149:0/1881961795 conn(0x7f1f440f09a0 msgr2=0x7f1f440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:20.849 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.847+0000 7f1f490cc700 1 -- 172.21.15.149:0/1881961795 shutdown_connections 2024-02-12T16:34:20.849 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.847+0000 7f1f490cc700 1 -- 172.21.15.149:0/1881961795 wait complete. 2024-02-12T16:34:20.851 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.848+0000 7f1f490cc700 1 Processor -- start 2024-02-12T16:34:20.851 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.848+0000 7f1f490cc700 1 -- start start 2024-02-12T16:34:20.851 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.848+0000 7f1f490cc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f440f51a0 0x7f1f441054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:20.851 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.848+0000 7f1f490cc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f441059c0 con 0x7f1f440f51a0 2024-02-12T16:34:20.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.848+0000 7f1f43fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f440f51a0 0x7f1f441054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:20.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.848+0000 7f1f43fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f440f51a0 0x7f1f441054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:52162/0 (socket says 172.21.15.149:52162) 2024-02-12T16:34:20.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.848+0000 7f1f43fff700 1 -- 172.21.15.149:0/2486913910 learned_addr learned my addr 172.21.15.149:0/2486913910 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:20.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.849+0000 7f1f43fff700 1 -- 172.21.15.149:0/2486913910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c005040 con 0x7f1f440f51a0 2024-02-12T16:34:20.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.849+0000 7f1f43fff700 1 --2- 172.21.15.149:0/2486913910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f440f51a0 0x7f1f441054d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f1f2c009830 tx=0x7f1f2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:20.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.849+0000 7f1f40ff9700 1 -- 172.21.15.149:0/2486913910 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f2c009320 con 0x7f1f440f51a0 2024-02-12T16:34:20.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.849+0000 7f1f490cc700 1 -- 172.21.15.149:0/2486913910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f44105bc0 con 0x7f1f440f51a0 2024-02-12T16:34:20.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.849+0000 7f1f40ff9700 1 -- 172.21.15.149:0/2486913910 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f2c013380 con 0x7f1f440f51a0 2024-02-12T16:34:20.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.849+0000 7f1f40ff9700 1 -- 172.21.15.149:0/2486913910 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f2c014420 con 0x7f1f440f51a0 2024-02-12T16:34:20.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.849+0000 7f1f490cc700 1 -- 172.21.15.149:0/2486913910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f44106050 con 0x7f1f440f51a0 2024-02-12T16:34:20.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.850+0000 7f1f40ff9700 1 -- 172.21.15.149:0/2486913910 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f2c01e420 con 0x7f1f440f51a0 2024-02-12T16:34:20.853 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.850+0000 7f1f490cc700 1 -- 172.21.15.149:0/2486913910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f44045ad0 con 0x7f1f440f51a0 2024-02-12T16:34:20.853 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.850+0000 7f1f40ff9700 1 --2- 172.21.15.149:0/2486913910 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f3002ffb0 0x7f1f30032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:20.853 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.850+0000 7f1f40ff9700 1 -- 172.21.15.149:0/2486913910 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f2c040df0 con 0x7f1f440f51a0 2024-02-12T16:34:20.853 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.850+0000 7f1f437fe700 1 --2- 172.21.15.149:0/2486913910 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f3002ffb0 0x7f1f30032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:20.853 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.851+0000 7f1f437fe700 1 --2- 172.21.15.149:0/2486913910 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f3002ffb0 0x7f1f30032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1f340096a0 tx=0x7f1f34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:20.855 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:20.854+0000 7f1f40ff9700 1 -- 172.21.15.149:0/2486913910 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f2c0095f0 con 0x7f1f440f51a0 2024-02-12T16:34:21.063 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.062+0000 7f1f490cc700 1 -- 172.21.15.149:0/2486913910 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f440fad30 con 0x7f1f440f51a0 2024-02-12T16:34:21.063 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.062+0000 7f1f40ff9700 1 -- 172.21.15.149:0/2486913910 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1f2c01cb10 con 0x7f1f440f51a0 2024-02-12T16:34:21.064 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:21.064 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:21.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.064+0000 7f1f490cc700 1 -- 172.21.15.149:0/2486913910 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f3002ffb0 msgr2=0x7f1f30032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:21.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.064+0000 7f1f490cc700 1 --2- 172.21.15.149:0/2486913910 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f3002ffb0 0x7f1f30032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1f340096a0 tx=0x7f1f34006b40).stop 2024-02-12T16:34:21.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.064+0000 7f1f490cc700 1 -- 172.21.15.149:0/2486913910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f440f51a0 msgr2=0x7f1f441054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:21.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.064+0000 7f1f490cc700 1 --2- 172.21.15.149:0/2486913910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f440f51a0 0x7f1f441054d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f1f2c009830 tx=0x7f1f2c00b040).stop 2024-02-12T16:34:21.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.064+0000 7f1f490cc700 1 -- 172.21.15.149:0/2486913910 shutdown_connections 2024-02-12T16:34:21.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.064+0000 7f1f490cc700 1 --2- 172.21.15.149:0/2486913910 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1f3002ffb0 0x7f1f30032470 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:21.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.064+0000 7f1f490cc700 1 --2- 172.21.15.149:0/2486913910 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f440f51a0 0x7f1f441054d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:21.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.064+0000 7f1f490cc700 1 -- 172.21.15.149:0/2486913910 >> 172.21.15.149:0/2486913910 conn(0x7f1f440f09a0 msgr2=0x7f1f440f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:21.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.064+0000 7f1f490cc700 1 -- 172.21.15.149:0/2486913910 shutdown_connections 2024-02-12T16:34:21.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:21.064+0000 7f1f490cc700 1 -- 172.21.15.149:0/2486913910 wait complete. 2024-02-12T16:34:21.066 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:21.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:21 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:21.062686+0000 mon.smithi104 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.149:0/2486913910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:22.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:20.318066+0000 mgr.smithi104.ohwbhf (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:22.706 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:22.706 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:24.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:22.318403+0000 mgr.smithi104.ohwbhf (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:24.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.577+0000 7fdd09618700 1 -- 172.21.15.149:0/2924114790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd040f51a0 msgr2=0x7fdd040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:24.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.577+0000 7fdd09618700 1 --2- 172.21.15.149:0/2924114790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd040f51a0 0x7fdd040f55c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fdcf8009fb0 tx=0x7fdcf8009b30).stop 2024-02-12T16:34:24.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.577+0000 7fdd09618700 1 -- 172.21.15.149:0/2924114790 shutdown_connections 2024-02-12T16:34:24.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.577+0000 7fdd09618700 1 --2- 172.21.15.149:0/2924114790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd040f51a0 0x7fdd040f55c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:24.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.577+0000 7fdd09618700 1 -- 172.21.15.149:0/2924114790 >> 172.21.15.149:0/2924114790 conn(0x7fdd040f09a0 msgr2=0x7fdd040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:24.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.577+0000 7fdd09618700 1 -- 172.21.15.149:0/2924114790 shutdown_connections 2024-02-12T16:34:24.580 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.577+0000 7fdd09618700 1 -- 172.21.15.149:0/2924114790 wait complete. 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.578+0000 7fdd09618700 1 Processor -- start 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.578+0000 7fdd09618700 1 -- start start 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.578+0000 7fdd09618700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd040f51a0 0x7fdd04109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.578+0000 7fdd09618700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04060280 con 0x7fdd040f51a0 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.578+0000 7fdd03fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd040f51a0 0x7fdd04109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.578+0000 7fdd03fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd040f51a0 0x7fdd04109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50786/0 (socket says 172.21.15.149:50786) 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.578+0000 7fdd03fff700 1 -- 172.21.15.149:0/3745157094 learned_addr learned my addr 172.21.15.149:0/3745157094 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.579+0000 7fdd03fff700 1 -- 172.21.15.149:0/3745157094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf8005040 con 0x7fdd040f51a0 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.579+0000 7fdd03fff700 1 --2- 172.21.15.149:0/3745157094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd040f51a0 0x7fdd04109290 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fdcf8009830 tx=0x7fdcf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.579+0000 7fdd00ff9700 1 -- 172.21.15.149:0/3745157094 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcf8009320 con 0x7fdd040f51a0 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.579+0000 7fdd09618700 1 -- 172.21.15.149:0/3745157094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd04060480 con 0x7fdd040f51a0 2024-02-12T16:34:24.581 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.579+0000 7fdd00ff9700 1 -- 172.21.15.149:0/3745157094 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcf8013380 con 0x7fdd040f51a0 2024-02-12T16:34:24.582 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.579+0000 7fdd00ff9700 1 -- 172.21.15.149:0/3745157094 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcf8014420 con 0x7fdd040f51a0 2024-02-12T16:34:24.582 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.579+0000 7fdd09618700 1 -- 172.21.15.149:0/3745157094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd041099c0 con 0x7fdd040f51a0 2024-02-12T16:34:24.582 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.580+0000 7fdd09618700 1 -- 172.21.15.149:0/3745157094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd04045ad0 con 0x7fdd040f51a0 2024-02-12T16:34:24.582 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.580+0000 7fdd00ff9700 1 -- 172.21.15.149:0/3745157094 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcf801e420 con 0x7fdd040f51a0 2024-02-12T16:34:24.582 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.580+0000 7fdd00ff9700 1 --2- 172.21.15.149:0/3745157094 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdcec02ff60 0x7fdcec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:24.583 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.580+0000 7fdd00ff9700 1 -- 172.21.15.149:0/3745157094 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcf8040e20 con 0x7fdd040f51a0 2024-02-12T16:34:24.583 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.580+0000 7fdd037fe700 1 --2- 172.21.15.149:0/3745157094 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdcec02ff60 0x7fdcec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:24.583 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.581+0000 7fdd037fe700 1 --2- 172.21.15.149:0/3745157094 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdcec02ff60 0x7fdcec032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdcf40096a0 tx=0x7fdcf4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:24.585 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.585+0000 7fdd00ff9700 1 -- 172.21.15.149:0/3745157094 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcf801c930 con 0x7fdd040f51a0 2024-02-12T16:34:24.788 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.787+0000 7fdd09618700 1 -- 172.21.15.149:0/3745157094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd0410a220 con 0x7fdd040f51a0 2024-02-12T16:34:24.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.788+0000 7fdd00ff9700 1 -- 172.21.15.149:0/3745157094 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdcf802c2d0 con 0x7fdd040f51a0 2024-02-12T16:34:24.789 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:24.789 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:24.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.789+0000 7fdd09618700 1 -- 172.21.15.149:0/3745157094 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdcec02ff60 msgr2=0x7fdcec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:24.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.789+0000 7fdd09618700 1 --2- 172.21.15.149:0/3745157094 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdcec02ff60 0x7fdcec032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdcf40096a0 tx=0x7fdcf4006b40).stop 2024-02-12T16:34:24.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.789+0000 7fdd09618700 1 -- 172.21.15.149:0/3745157094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd040f51a0 msgr2=0x7fdd04109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:24.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.789+0000 7fdd09618700 1 --2- 172.21.15.149:0/3745157094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd040f51a0 0x7fdd04109290 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fdcf8009830 tx=0x7fdcf800b040).stop 2024-02-12T16:34:24.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.790+0000 7fdd09618700 1 -- 172.21.15.149:0/3745157094 shutdown_connections 2024-02-12T16:34:24.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.790+0000 7fdd09618700 1 --2- 172.21.15.149:0/3745157094 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdcec02ff60 0x7fdcec032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:24.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.790+0000 7fdd09618700 1 --2- 172.21.15.149:0/3745157094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd040f51a0 0x7fdd04109290 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:24.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.790+0000 7fdd09618700 1 -- 172.21.15.149:0/3745157094 >> 172.21.15.149:0/3745157094 conn(0x7fdd040f09a0 msgr2=0x7fdd040f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:24.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.790+0000 7fdd09618700 1 -- 172.21.15.149:0/3745157094 shutdown_connections 2024-02-12T16:34:24.791 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:24.790+0000 7fdd09618700 1 -- 172.21.15.149:0/3745157094 wait complete. 2024-02-12T16:34:24.792 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:24.406765+0000 mon.smithi104 (mon.0) 196 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:34:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:24.413510+0000 mon.smithi104 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:34:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:24.788230+0000 mon.smithi104 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.149:0/3745157094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:26.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:24.318727+0000 mgr.smithi104.ohwbhf (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:26.439 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:26.440 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:28.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.169+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3621363268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbbf40f5410 msgr2=0x7fbbf40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:28.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.169+0000 7fbbfbdec700 1 --2- 172.21.15.149:0/3621363268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbbf40f5410 0x7fbbf40f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fbbe4009fb0 tx=0x7fbbe4009b30).stop 2024-02-12T16:34:28.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.169+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3621363268 shutdown_connections 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.169+0000 7fbbfbdec700 1 --2- 172.21.15.149:0/3621363268 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbbf40f5410 0x7fbbf40f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.169+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3621363268 >> 172.21.15.149:0/3621363268 conn(0x7fbbf40f0b70 msgr2=0x7fbbf40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.169+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3621363268 shutdown_connections 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.169+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3621363268 wait complete. 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.170+0000 7fbbfbdec700 1 Processor -- start 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.170+0000 7fbbfbdec700 1 -- start start 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.170+0000 7fbbfbdec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbbf40f5410 0x7fbbf41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.170+0000 7fbbfbdec700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf4106cd0 con 0x7fbbf40f5410 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.170+0000 7fbbf9b88700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbbf40f5410 0x7fbbf41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.170+0000 7fbbf9b88700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbbf40f5410 0x7fbbf41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50802/0 (socket says 172.21.15.149:50802) 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.170+0000 7fbbf9b88700 1 -- 172.21.15.149:0/3892772192 learned_addr learned my addr 172.21.15.149:0/3892772192 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:28.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.170+0000 7fbbf9b88700 1 -- 172.21.15.149:0/3892772192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe4005040 con 0x7fbbf40f5410 2024-02-12T16:34:28.175 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.171+0000 7fbbf9b88700 1 --2- 172.21.15.149:0/3892772192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbbf40f5410 0x7fbbf41067e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fbbe4005010 tx=0x7fbbe400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:28.175 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.171+0000 7fbbea7fc700 1 -- 172.21.15.149:0/3892772192 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbe4009320 con 0x7fbbf40f5410 2024-02-12T16:34:28.175 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.171+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3892772192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf4106ed0 con 0x7fbbf40f5410 2024-02-12T16:34:28.175 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.171+0000 7fbbea7fc700 1 -- 172.21.15.149:0/3892772192 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbe4013380 con 0x7fbbf40f5410 2024-02-12T16:34:28.175 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.171+0000 7fbbea7fc700 1 -- 172.21.15.149:0/3892772192 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbe4014e50 con 0x7fbbf40f5410 2024-02-12T16:34:28.175 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.171+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3892772192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf4107360 con 0x7fbbf40f5410 2024-02-12T16:34:28.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.172+0000 7fbbea7fc700 1 -- 172.21.15.149:0/3892772192 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbbe4014420 con 0x7fbbf40f5410 2024-02-12T16:34:28.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.172+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3892772192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf4045ad0 con 0x7fbbf40f5410 2024-02-12T16:34:28.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.172+0000 7fbbea7fc700 1 --2- 172.21.15.149:0/3892772192 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbbe002ff10 0x7fbbe00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:28.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.172+0000 7fbbea7fc700 1 -- 172.21.15.149:0/3892772192 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbbe4040e30 con 0x7fbbf40f5410 2024-02-12T16:34:28.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.172+0000 7fbbf9387700 1 --2- 172.21.15.149:0/3892772192 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbbe002ff10 0x7fbbe00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:28.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.173+0000 7fbbf9387700 1 --2- 172.21.15.149:0/3892772192 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbbe002ff10 0x7fbbe00323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbbf00096a0 tx=0x7fbbf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:28.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.177+0000 7fbbea7fc700 1 -- 172.21.15.149:0/3892772192 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbe40095f0 con 0x7fbbf40f5410 2024-02-12T16:34:28.385 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.383+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3892772192 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbf40fafa0 con 0x7fbbf40f5410 2024-02-12T16:34:28.386 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.384+0000 7fbbea7fc700 1 -- 172.21.15.149:0/3892772192 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbbe401db70 con 0x7fbbf40f5410 2024-02-12T16:34:28.386 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:28.386 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:28.387 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.385+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3892772192 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbbe002ff10 msgr2=0x7fbbe00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:28.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.385+0000 7fbbfbdec700 1 --2- 172.21.15.149:0/3892772192 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbbe002ff10 0x7fbbe00323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbbf00096a0 tx=0x7fbbf0006b40).stop 2024-02-12T16:34:28.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.385+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3892772192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbbf40f5410 msgr2=0x7fbbf41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:28.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.385+0000 7fbbfbdec700 1 --2- 172.21.15.149:0/3892772192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbbf40f5410 0x7fbbf41067e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fbbe4005010 tx=0x7fbbe400b040).stop 2024-02-12T16:34:28.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.385+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3892772192 shutdown_connections 2024-02-12T16:34:28.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.385+0000 7fbbfbdec700 1 --2- 172.21.15.149:0/3892772192 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fbbe002ff10 0x7fbbe00323d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:28.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.385+0000 7fbbfbdec700 1 --2- 172.21.15.149:0/3892772192 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbbf40f5410 0x7fbbf41067e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:28.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.385+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3892772192 >> 172.21.15.149:0/3892772192 conn(0x7fbbf40f0b70 msgr2=0x7fbbf405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:28.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.385+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3892772192 shutdown_connections 2024-02-12T16:34:28.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:28.385+0000 7fbbfbdec700 1 -- 172.21.15.149:0/3892772192 wait complete. 2024-02-12T16:34:28.389 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:28.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:26.319136+0000 mgr.smithi104.ohwbhf (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:29.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:29 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:28.384839+0000 mon.smithi104 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.149:0/3892772192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:30.040 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:30.041 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:30.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:28.319393+0000 mgr.smithi104.ohwbhf (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:31.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.809+0000 7f6d8a081700 1 -- 172.21.15.149:0/729428514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d84060280 msgr2=0x7f6d840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:31.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.809+0000 7f6d8a081700 1 --2- 172.21.15.149:0/729428514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d84060280 0x7f6d840f90f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f6d74009fb0 tx=0x7f6d74009b30).stop 2024-02-12T16:34:31.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.809+0000 7f6d8a081700 1 -- 172.21.15.149:0/729428514 shutdown_connections 2024-02-12T16:34:31.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.809+0000 7f6d8a081700 1 --2- 172.21.15.149:0/729428514 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d84060280 0x7f6d840f90f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:31.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.809+0000 7f6d8a081700 1 -- 172.21.15.149:0/729428514 >> 172.21.15.149:0/729428514 conn(0x7f6d840f0b70 msgr2=0x7f6d840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:31.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.809+0000 7f6d8a081700 1 -- 172.21.15.149:0/729428514 shutdown_connections 2024-02-12T16:34:31.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.809+0000 7f6d8a081700 1 -- 172.21.15.149:0/729428514 wait complete. 2024-02-12T16:34:31.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.810+0000 7f6d8a081700 1 Processor -- start 2024-02-12T16:34:31.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.810+0000 7f6d8a081700 1 -- start start 2024-02-12T16:34:31.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.810+0000 7f6d8a081700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d84060280 0x7f6d840f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:31.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.810+0000 7f6d8a081700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d840f4ff0 con 0x7f6d84060280 2024-02-12T16:34:31.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.810+0000 7f6d837fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d84060280 0x7f6d840f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:31.815 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.810+0000 7f6d837fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d84060280 0x7f6d840f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50808/0 (socket says 172.21.15.149:50808) 2024-02-12T16:34:31.815 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.810+0000 7f6d837fe700 1 -- 172.21.15.149:0/1681825625 learned_addr learned my addr 172.21.15.149:0/1681825625 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:31.815 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.811+0000 7f6d837fe700 1 -- 172.21.15.149:0/1681825625 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d74005040 con 0x7f6d84060280 2024-02-12T16:34:31.815 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.811+0000 7f6d837fe700 1 --2- 172.21.15.149:0/1681825625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d84060280 0x7f6d840f6b70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f6d740089f0 tx=0x7f6d7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:31.815 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.811+0000 7f6d7bfff700 1 -- 172.21.15.149:0/1681825625 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d74009320 con 0x7f6d84060280 2024-02-12T16:34:31.815 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.811+0000 7f6d8a081700 1 -- 172.21.15.149:0/1681825625 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d840f5190 con 0x7f6d84060280 2024-02-12T16:34:31.815 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.811+0000 7f6d7bfff700 1 -- 172.21.15.149:0/1681825625 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d74013380 con 0x7f6d84060280 2024-02-12T16:34:31.815 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.811+0000 7f6d7bfff700 1 -- 172.21.15.149:0/1681825625 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d74014e20 con 0x7f6d84060280 2024-02-12T16:34:31.815 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.811+0000 7f6d8a081700 1 -- 172.21.15.149:0/1681825625 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d840f5620 con 0x7f6d84060280 2024-02-12T16:34:31.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.812+0000 7f6d7bfff700 1 -- 172.21.15.149:0/1681825625 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d740095f0 con 0x7f6d84060280 2024-02-12T16:34:31.817 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.812+0000 7f6d8a081700 1 -- 172.21.15.149:0/1681825625 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d84059100 con 0x7f6d84060280 2024-02-12T16:34:31.817 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.812+0000 7f6d7bfff700 1 --2- 172.21.15.149:0/1681825625 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7002ff10 0x7f6d700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:31.817 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.812+0000 7f6d7bfff700 1 -- 172.21.15.149:0/1681825625 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d74040f50 con 0x7f6d84060280 2024-02-12T16:34:31.817 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.812+0000 7f6d82ffd700 1 --2- 172.21.15.149:0/1681825625 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7002ff10 0x7f6d700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:31.817 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.812+0000 7f6d82ffd700 1 --2- 172.21.15.149:0/1681825625 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7002ff10 0x7f6d700323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f6d6c0096a0 tx=0x7f6d6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:31.818 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:31.816+0000 7f6d7bfff700 1 -- 172.21.15.149:0/1681825625 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d74014420 con 0x7f6d84060280 2024-02-12T16:34:32.023 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.021+0000 7f6d8a081700 1 -- 172.21.15.149:0/1681825625 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d840fc090 con 0x7f6d84060280 2024-02-12T16:34:32.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.022+0000 7f6d7bfff700 1 -- 172.21.15.149:0/1681825625 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6d7401d980 con 0x7f6d84060280 2024-02-12T16:34:32.024 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:32.024 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:32.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.024+0000 7f6d8a081700 1 -- 172.21.15.149:0/1681825625 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7002ff10 msgr2=0x7f6d700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:32.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.024+0000 7f6d8a081700 1 --2- 172.21.15.149:0/1681825625 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7002ff10 0x7f6d700323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f6d6c0096a0 tx=0x7f6d6c006b40).stop 2024-02-12T16:34:32.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.024+0000 7f6d8a081700 1 -- 172.21.15.149:0/1681825625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d84060280 msgr2=0x7f6d840f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:32.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.024+0000 7f6d8a081700 1 --2- 172.21.15.149:0/1681825625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d84060280 0x7f6d840f6b70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f6d740089f0 tx=0x7f6d7400b040).stop 2024-02-12T16:34:32.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.024+0000 7f6d8a081700 1 -- 172.21.15.149:0/1681825625 shutdown_connections 2024-02-12T16:34:32.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.024+0000 7f6d8a081700 1 --2- 172.21.15.149:0/1681825625 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6d7002ff10 0x7f6d700323d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:32.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.024+0000 7f6d8a081700 1 --2- 172.21.15.149:0/1681825625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d84060280 0x7f6d840f6b70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:32.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.024+0000 7f6d8a081700 1 -- 172.21.15.149:0/1681825625 >> 172.21.15.149:0/1681825625 conn(0x7f6d840f0b70 msgr2=0x7f6d840f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:32.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.024+0000 7f6d8a081700 1 -- 172.21.15.149:0/1681825625 shutdown_connections 2024-02-12T16:34:32.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:32.024+0000 7f6d8a081700 1 -- 172.21.15.149:0/1681825625 wait complete. 2024-02-12T16:34:32.027 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:32.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:30.319714+0000 mgr.smithi104.ohwbhf (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:32.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:32.023242+0000 mon.smithi104 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.149:0/1681825625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:33.686 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:33.686 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:34.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:32.320091+0000 mgr.smithi104.ohwbhf (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:35.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.431+0000 7f83dd725700 1 -- 172.21.15.149:0/1111691925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83d80f51a0 msgr2=0x7f83d80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:35.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.431+0000 7f83dd725700 1 --2- 172.21.15.149:0/1111691925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83d80f51a0 0x7f83d80f55c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f83c8009fb0 tx=0x7f83c8009b30).stop 2024-02-12T16:34:35.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.431+0000 7f83dd725700 1 -- 172.21.15.149:0/1111691925 shutdown_connections 2024-02-12T16:34:35.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.431+0000 7f83dd725700 1 --2- 172.21.15.149:0/1111691925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83d80f51a0 0x7f83d80f55c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:35.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.431+0000 7f83dd725700 1 -- 172.21.15.149:0/1111691925 >> 172.21.15.149:0/1111691925 conn(0x7f83d80f09a0 msgr2=0x7f83d80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:35.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.431+0000 7f83dd725700 1 -- 172.21.15.149:0/1111691925 shutdown_connections 2024-02-12T16:34:35.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.431+0000 7f83dd725700 1 -- 172.21.15.149:0/1111691925 wait complete. 2024-02-12T16:34:35.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.432+0000 7f83dd725700 1 Processor -- start 2024-02-12T16:34:35.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.432+0000 7f83dd725700 1 -- start start 2024-02-12T16:34:35.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.432+0000 7f83dd725700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83d80f51a0 0x7f83d80f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.432+0000 7f83dd725700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d80f13d0 con 0x7f83d80f51a0 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.432+0000 7f83d7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83d80f51a0 0x7f83d80f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.432+0000 7f83d7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83d80f51a0 0x7f83d80f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:40722/0 (socket says 172.21.15.149:40722) 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.432+0000 7f83d7fff700 1 -- 172.21.15.149:0/3040056005 learned_addr learned my addr 172.21.15.149:0/3040056005 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.432+0000 7f83d7fff700 1 -- 172.21.15.149:0/3040056005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c8005040 con 0x7f83d80f51a0 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.432+0000 7f83d7fff700 1 --2- 172.21.15.149:0/3040056005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83d80f51a0 0x7f83d80f2f50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f83c8009f80 tx=0x7f83c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.433+0000 7f83d57fa700 1 -- 172.21.15.149:0/3040056005 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83c80095f0 con 0x7f83d80f51a0 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.433+0000 7f83dd725700 1 -- 172.21.15.149:0/3040056005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d80f15d0 con 0x7f83d80f51a0 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.433+0000 7f83d57fa700 1 -- 172.21.15.149:0/3040056005 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83c8007e90 con 0x7f83d80f51a0 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.433+0000 7f83d57fa700 1 -- 172.21.15.149:0/3040056005 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83c8014d80 con 0x7f83d80f51a0 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.433+0000 7f83dd725700 1 -- 172.21.15.149:0/3040056005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d80f1a60 con 0x7f83d80f51a0 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.433+0000 7f83d57fa700 1 -- 172.21.15.149:0/3040056005 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83c8025070 con 0x7f83d80f51a0 2024-02-12T16:34:35.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.434+0000 7f83d57fa700 1 --2- 172.21.15.149:0/3040056005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f83c402ff60 0x7f83c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:35.466 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.434+0000 7f83d57fa700 1 -- 172.21.15.149:0/3040056005 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83c8040960 con 0x7f83d80f51a0 2024-02-12T16:34:35.466 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.434+0000 7f83dd725700 1 -- 172.21.15.149:0/3040056005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d8059100 con 0x7f83d80f51a0 2024-02-12T16:34:35.466 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.434+0000 7f83cffff700 1 --2- 172.21.15.149:0/3040056005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f83c402ff60 0x7f83c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:35.466 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.434+0000 7f83cffff700 1 --2- 172.21.15.149:0/3040056005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f83c402ff60 0x7f83c4032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f83c00096a0 tx=0x7f83c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:35.466 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.438+0000 7f83d57fa700 1 -- 172.21.15.149:0/3040056005 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83c800f300 con 0x7f83d80f51a0 2024-02-12T16:34:35.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.643+0000 7f83dd725700 1 -- 172.21.15.149:0/3040056005 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83d8027a00 con 0x7f83d80f51a0 2024-02-12T16:34:35.646 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.644+0000 7f83d57fa700 1 -- 172.21.15.149:0/3040056005 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f83c8025af0 con 0x7f83d80f51a0 2024-02-12T16:34:35.646 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:35.646 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:35.648 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.646+0000 7f83dd725700 1 -- 172.21.15.149:0/3040056005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f83c402ff60 msgr2=0x7f83c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:35.648 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.646+0000 7f83dd725700 1 --2- 172.21.15.149:0/3040056005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f83c402ff60 0x7f83c4032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f83c00096a0 tx=0x7f83c0006b40).stop 2024-02-12T16:34:35.648 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.646+0000 7f83dd725700 1 -- 172.21.15.149:0/3040056005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83d80f51a0 msgr2=0x7f83d80f2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:35.648 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.646+0000 7f83dd725700 1 --2- 172.21.15.149:0/3040056005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83d80f51a0 0x7f83d80f2f50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f83c8009f80 tx=0x7f83c800b040).stop 2024-02-12T16:34:35.648 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.646+0000 7f83dd725700 1 -- 172.21.15.149:0/3040056005 shutdown_connections 2024-02-12T16:34:35.648 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.646+0000 7f83dd725700 1 --2- 172.21.15.149:0/3040056005 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f83c402ff60 0x7f83c4032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:35.648 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.646+0000 7f83dd725700 1 --2- 172.21.15.149:0/3040056005 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83d80f51a0 0x7f83d80f2f50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:35.648 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.646+0000 7f83dd725700 1 -- 172.21.15.149:0/3040056005 >> 172.21.15.149:0/3040056005 conn(0x7f83d80f09a0 msgr2=0x7f83d805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:35.648 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.646+0000 7f83dd725700 1 -- 172.21.15.149:0/3040056005 shutdown_connections 2024-02-12T16:34:35.649 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:35.646+0000 7f83dd725700 1 -- 172.21.15.149:0/3040056005 wait complete. 2024-02-12T16:34:35.649 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:36.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:34.320441+0000 mgr.smithi104.ohwbhf (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:36.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:35.645217+0000 mon.smithi104 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.149:0/3040056005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:37.304 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:37.305 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:38.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:36.320723+0000 mgr.smithi104.ohwbhf (mgr.14162) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:39.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.090+0000 7f730359e700 1 -- 172.21.15.149:0/4068808359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73040f51a0 msgr2=0x7f73040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:39.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.090+0000 7f730359e700 1 --2- 172.21.15.149:0/4068808359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73040f51a0 0x7f73040f55c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f72f4009fb0 tx=0x7f72f4009b30).stop 2024-02-12T16:34:39.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.090+0000 7f730359e700 1 -- 172.21.15.149:0/4068808359 shutdown_connections 2024-02-12T16:34:39.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.090+0000 7f730359e700 1 --2- 172.21.15.149:0/4068808359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73040f51a0 0x7f73040f55c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:39.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.090+0000 7f730359e700 1 -- 172.21.15.149:0/4068808359 >> 172.21.15.149:0/4068808359 conn(0x7f73040f09a0 msgr2=0x7f73040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:39.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.090+0000 7f730359e700 1 -- 172.21.15.149:0/4068808359 shutdown_connections 2024-02-12T16:34:39.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.090+0000 7f730359e700 1 -- 172.21.15.149:0/4068808359 wait complete. 2024-02-12T16:34:39.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.090+0000 7f730359e700 1 Processor -- start 2024-02-12T16:34:39.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.090+0000 7f730359e700 1 -- start start 2024-02-12T16:34:39.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f730359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73040f51a0 0x7f73041054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:39.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f730359e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73041059c0 con 0x7f73040f51a0 2024-02-12T16:34:39.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f730259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73040f51a0 0x7f73041054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:39.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f730259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73040f51a0 0x7f73041054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:40744/0 (socket says 172.21.15.149:40744) 2024-02-12T16:34:39.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f730259c700 1 -- 172.21.15.149:0/2412766692 learned_addr learned my addr 172.21.15.149:0/2412766692 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:39.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f730259c700 1 -- 172.21.15.149:0/2412766692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f4005040 con 0x7f73040f51a0 2024-02-12T16:34:39.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f730259c700 1 --2- 172.21.15.149:0/2412766692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73040f51a0 0x7f73041054d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f72f40089f0 tx=0x7f72f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:39.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f72f2ffd700 1 -- 172.21.15.149:0/2412766692 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72f4009320 con 0x7f73040f51a0 2024-02-12T16:34:39.099 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f730359e700 1 -- 172.21.15.149:0/2412766692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7304105bc0 con 0x7f73040f51a0 2024-02-12T16:34:39.099 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f72f2ffd700 1 -- 172.21.15.149:0/2412766692 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72f4013380 con 0x7f73040f51a0 2024-02-12T16:34:39.099 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f72f2ffd700 1 -- 172.21.15.149:0/2412766692 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72f4014e20 con 0x7f73040f51a0 2024-02-12T16:34:39.099 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.091+0000 7f730359e700 1 -- 172.21.15.149:0/2412766692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7304106050 con 0x7f73040f51a0 2024-02-12T16:34:39.099 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.092+0000 7f72f2ffd700 1 -- 172.21.15.149:0/2412766692 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72f40095f0 con 0x7f73040f51a0 2024-02-12T16:34:39.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.092+0000 7f730359e700 1 -- 172.21.15.149:0/2412766692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7304045ad0 con 0x7f73040f51a0 2024-02-12T16:34:39.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.092+0000 7f72f2ffd700 1 --2- 172.21.15.149:0/2412766692 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f72ec02ff10 0x7f72ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:39.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.092+0000 7f72f2ffd700 1 -- 172.21.15.149:0/2412766692 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72f4040f10 con 0x7f73040f51a0 2024-02-12T16:34:39.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.093+0000 7f7301d9b700 1 --2- 172.21.15.149:0/2412766692 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f72ec02ff10 0x7f72ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:39.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.093+0000 7f7301d9b700 1 --2- 172.21.15.149:0/2412766692 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f72ec02ff10 0x7f72ec0323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f72f80096a0 tx=0x7f72f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:39.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.097+0000 7f72f2ffd700 1 -- 172.21.15.149:0/2412766692 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72f4014420 con 0x7f73040f51a0 2024-02-12T16:34:39.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.301+0000 7f730359e700 1 -- 172.21.15.149:0/2412766692 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7304106f00 con 0x7f73040f51a0 2024-02-12T16:34:39.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.301+0000 7f72f2ffd700 1 -- 172.21.15.149:0/2412766692 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f72f401d980 con 0x7f73040f51a0 2024-02-12T16:34:39.304 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:39.304 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:39.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.303+0000 7f730359e700 1 -- 172.21.15.149:0/2412766692 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f72ec02ff10 msgr2=0x7f72ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:39.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.303+0000 7f730359e700 1 --2- 172.21.15.149:0/2412766692 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f72ec02ff10 0x7f72ec0323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f72f80096a0 tx=0x7f72f8006b40).stop 2024-02-12T16:34:39.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.303+0000 7f730359e700 1 -- 172.21.15.149:0/2412766692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73040f51a0 msgr2=0x7f73041054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:39.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.303+0000 7f730359e700 1 --2- 172.21.15.149:0/2412766692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73040f51a0 0x7f73041054d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f72f40089f0 tx=0x7f72f400b040).stop 2024-02-12T16:34:39.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.303+0000 7f730359e700 1 -- 172.21.15.149:0/2412766692 shutdown_connections 2024-02-12T16:34:39.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.303+0000 7f730359e700 1 --2- 172.21.15.149:0/2412766692 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f72ec02ff10 0x7f72ec0323d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:39.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.303+0000 7f730359e700 1 --2- 172.21.15.149:0/2412766692 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73040f51a0 0x7f73041054d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:39.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.303+0000 7f730359e700 1 -- 172.21.15.149:0/2412766692 >> 172.21.15.149:0/2412766692 conn(0x7f73040f09a0 msgr2=0x7f73040f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:39.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.303+0000 7f730359e700 1 -- 172.21.15.149:0/2412766692 shutdown_connections 2024-02-12T16:34:39.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:39.303+0000 7f730359e700 1 -- 172.21.15.149:0/2412766692 wait complete. 2024-02-12T16:34:39.306 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:40.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:38.320998+0000 mgr.smithi104.ohwbhf (mgr.14162) 158 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:40.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:39.302704+0000 mon.smithi104 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.149:0/2412766692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:40.915 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:40.916 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:42.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:40.321332+0000 mgr.smithi104.ohwbhf (mgr.14162) 159 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:42.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.771+0000 7fb93359e700 1 -- 172.21.15.149:0/3837235401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340f78e0 msgr2=0x7fb9340f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:42.774 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.771+0000 7fb93359e700 1 --2- 172.21.15.149:0/3837235401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340f78e0 0x7fb9340f7d00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fb924009fb0 tx=0x7fb924009b30).stop 2024-02-12T16:34:42.774 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.771+0000 7fb93359e700 1 -- 172.21.15.149:0/3837235401 shutdown_connections 2024-02-12T16:34:42.774 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.771+0000 7fb93359e700 1 --2- 172.21.15.149:0/3837235401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340f78e0 0x7fb9340f7d00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:42.774 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.771+0000 7fb93359e700 1 -- 172.21.15.149:0/3837235401 >> 172.21.15.149:0/3837235401 conn(0x7fb9340f0120 msgr2=0x7fb9340f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:42.774 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.771+0000 7fb93359e700 1 -- 172.21.15.149:0/3837235401 shutdown_connections 2024-02-12T16:34:42.774 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.771+0000 7fb93359e700 1 -- 172.21.15.149:0/3837235401 wait complete. 2024-02-12T16:34:42.774 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.771+0000 7fb93359e700 1 Processor -- start 2024-02-12T16:34:42.774 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.772+0000 7fb93359e700 1 -- start start 2024-02-12T16:34:42.774 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.772+0000 7fb93359e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340f78e0 0x7fb9341065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:42.775 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.772+0000 7fb93359e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb934106aa0 con 0x7fb9340f78e0 2024-02-12T16:34:42.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.772+0000 7fb93259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340f78e0 0x7fb9341065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:42.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.772+0000 7fb93259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340f78e0 0x7fb9341065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:40762/0 (socket says 172.21.15.149:40762) 2024-02-12T16:34:42.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.772+0000 7fb93259c700 1 -- 172.21.15.149:0/3779310120 learned_addr learned my addr 172.21.15.149:0/3779310120 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.772+0000 7fb93259c700 1 -- 172.21.15.149:0/3779310120 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb924005040 con 0x7fb9340f78e0 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.772+0000 7fb93259c700 1 --2- 172.21.15.149:0/3779310120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340f78e0 0x7fb9341065b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fb9240089f0 tx=0x7fb92400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.773+0000 7fb922ffd700 1 -- 172.21.15.149:0/3779310120 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb924009320 con 0x7fb9340f78e0 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.773+0000 7fb922ffd700 1 -- 172.21.15.149:0/3779310120 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb924013380 con 0x7fb9340f78e0 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.773+0000 7fb93359e700 1 -- 172.21.15.149:0/3779310120 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb934106ca0 con 0x7fb9340f78e0 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.773+0000 7fb922ffd700 1 -- 172.21.15.149:0/3779310120 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb924014e20 con 0x7fb9340f78e0 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.773+0000 7fb93359e700 1 -- 172.21.15.149:0/3779310120 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb934107130 con 0x7fb9340f78e0 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.773+0000 7fb922ffd700 1 -- 172.21.15.149:0/3779310120 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9240095f0 con 0x7fb9340f78e0 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.773+0000 7fb93359e700 1 -- 172.21.15.149:0/3779310120 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb934045ad0 con 0x7fb9340f78e0 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.774+0000 7fb922ffd700 1 --2- 172.21.15.149:0/3779310120 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb91c02ff10 0x7fb91c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.774+0000 7fb922ffd700 1 -- 172.21.15.149:0/3779310120 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb924040ed0 con 0x7fb9340f78e0 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.774+0000 7fb931d9b700 1 --2- 172.21.15.149:0/3779310120 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb91c02ff10 0x7fb91c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:42.777 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.774+0000 7fb931d9b700 1 --2- 172.21.15.149:0/3779310120 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb91c02ff10 0x7fb91c0323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fb9280096a0 tx=0x7fb928006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:42.780 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.778+0000 7fb922ffd700 1 -- 172.21.15.149:0/3779310120 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb924014420 con 0x7fb9340f78e0 2024-02-12T16:34:42.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.981+0000 7fb93359e700 1 -- 172.21.15.149:0/3779310120 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb934107fe0 con 0x7fb9340f78e0 2024-02-12T16:34:42.983 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.981+0000 7fb922ffd700 1 -- 172.21.15.149:0/3779310120 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb92401d980 con 0x7fb9340f78e0 2024-02-12T16:34:42.983 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:42.984 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:42.985 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.983+0000 7fb93359e700 1 -- 172.21.15.149:0/3779310120 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb91c02ff10 msgr2=0x7fb91c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:42.985 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.983+0000 7fb93359e700 1 --2- 172.21.15.149:0/3779310120 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb91c02ff10 0x7fb91c0323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fb9280096a0 tx=0x7fb928006b40).stop 2024-02-12T16:34:42.985 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.983+0000 7fb93359e700 1 -- 172.21.15.149:0/3779310120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340f78e0 msgr2=0x7fb9341065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:42.985 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.983+0000 7fb93359e700 1 --2- 172.21.15.149:0/3779310120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340f78e0 0x7fb9341065b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fb9240089f0 tx=0x7fb92400b040).stop 2024-02-12T16:34:42.985 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.983+0000 7fb93359e700 1 -- 172.21.15.149:0/3779310120 shutdown_connections 2024-02-12T16:34:42.985 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.983+0000 7fb93359e700 1 --2- 172.21.15.149:0/3779310120 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb91c02ff10 0x7fb91c0323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:42.985 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.983+0000 7fb93359e700 1 --2- 172.21.15.149:0/3779310120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9340f78e0 0x7fb9341065b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:42.985 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.983+0000 7fb93359e700 1 -- 172.21.15.149:0/3779310120 >> 172.21.15.149:0/3779310120 conn(0x7fb9340f0120 msgr2=0x7fb9340f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:42.985 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.983+0000 7fb93359e700 1 -- 172.21.15.149:0/3779310120 shutdown_connections 2024-02-12T16:34:42.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:42.983+0000 7fb93359e700 1 -- 172.21.15.149:0/3779310120 wait complete. 2024-02-12T16:34:42.986 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:43.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:43 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:42.982585+0000 mon.smithi104 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.149:0/3779310120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:44.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:42.321667+0000 mgr.smithi104.ohwbhf (mgr.14162) 160 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:44.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:44.510 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:46.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.307+0000 7fc7982b7700 1 -- 172.21.15.149:0/1068078955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7900f5410 msgr2=0x7fc7900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:46.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.307+0000 7fc7982b7700 1 --2- 172.21.15.149:0/1068078955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7900f5410 0x7fc7900f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fc780009fb0 tx=0x7fc780009b30).stop 2024-02-12T16:34:46.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.307+0000 7fc7982b7700 1 -- 172.21.15.149:0/1068078955 shutdown_connections 2024-02-12T16:34:46.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.307+0000 7fc7982b7700 1 --2- 172.21.15.149:0/1068078955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7900f5410 0x7fc7900f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:46.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.307+0000 7fc7982b7700 1 -- 172.21.15.149:0/1068078955 >> 172.21.15.149:0/1068078955 conn(0x7fc7900f0b70 msgr2=0x7fc7900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:46.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.307+0000 7fc7982b7700 1 -- 172.21.15.149:0/1068078955 shutdown_connections 2024-02-12T16:34:46.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.307+0000 7fc7982b7700 1 -- 172.21.15.149:0/1068078955 wait complete. 2024-02-12T16:34:46.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.307+0000 7fc7982b7700 1 Processor -- start 2024-02-12T16:34:46.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.308+0000 7fc7982b7700 1 -- start start 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.308+0000 7fc7982b7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7900f5410 0x7fc7900fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.308+0000 7fc7982b7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7900fb960 con 0x7fc7900f5410 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.308+0000 7fc796053700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7900f5410 0x7fc7900fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.308+0000 7fc796053700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7900f5410 0x7fc7900fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33180/0 (socket says 172.21.15.149:33180) 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.308+0000 7fc796053700 1 -- 172.21.15.149:0/883257348 learned_addr learned my addr 172.21.15.149:0/883257348 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.308+0000 7fc796053700 1 -- 172.21.15.149:0/883257348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc780005040 con 0x7fc7900f5410 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.308+0000 7fc796053700 1 --2- 172.21.15.149:0/883257348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7900f5410 0x7fc7900fd4e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fc780005010 tx=0x7fc78000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.308+0000 7fc786ffd700 1 -- 172.21.15.149:0/883257348 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc780009320 con 0x7fc7900f5410 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.308+0000 7fc7982b7700 1 -- 172.21.15.149:0/883257348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7900fbb00 con 0x7fc7900f5410 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.308+0000 7fc786ffd700 1 -- 172.21.15.149:0/883257348 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc780013380 con 0x7fc7900f5410 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.309+0000 7fc786ffd700 1 -- 172.21.15.149:0/883257348 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc780014e50 con 0x7fc7900f5410 2024-02-12T16:34:46.313 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.309+0000 7fc7982b7700 1 -- 172.21.15.149:0/883257348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7900fbf90 con 0x7fc7900f5410 2024-02-12T16:34:46.314 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.309+0000 7fc786ffd700 1 -- 172.21.15.149:0/883257348 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7800263a0 con 0x7fc7900f5410 2024-02-12T16:34:46.314 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.309+0000 7fc786ffd700 1 --2- 172.21.15.149:0/883257348 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc77c02ff10 0x7fc77c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:46.314 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.309+0000 7fc786ffd700 1 -- 172.21.15.149:0/883257348 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc780040fa0 con 0x7fc7900f5410 2024-02-12T16:34:46.314 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.310+0000 7fc7982b7700 1 -- 172.21.15.149:0/883257348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc790059100 con 0x7fc7900f5410 2024-02-12T16:34:46.314 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.310+0000 7fc795852700 1 --2- 172.21.15.149:0/883257348 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc77c02ff10 0x7fc77c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:46.314 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.310+0000 7fc795852700 1 --2- 172.21.15.149:0/883257348 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc77c02ff10 0x7fc77c0323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fc78c0096a0 tx=0x7fc78c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:46.316 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.314+0000 7fc786ffd700 1 -- 172.21.15.149:0/883257348 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc780012920 con 0x7fc7900f5410 2024-02-12T16:34:46.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:44.322055+0000 mgr.smithi104.ohwbhf (mgr.14162) 161 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:46.520 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.518+0000 7fc7982b7700 1 -- 172.21.15.149:0/883257348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7900fc690 con 0x7fc7900f5410 2024-02-12T16:34:46.521 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.519+0000 7fc786ffd700 1 -- 172.21.15.149:0/883257348 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc780012330 con 0x7fc7900f5410 2024-02-12T16:34:46.521 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:46.522 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:46.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.521+0000 7fc7982b7700 1 -- 172.21.15.149:0/883257348 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc77c02ff10 msgr2=0x7fc77c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:46.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.521+0000 7fc7982b7700 1 --2- 172.21.15.149:0/883257348 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc77c02ff10 0x7fc77c0323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fc78c0096a0 tx=0x7fc78c006b40).stop 2024-02-12T16:34:46.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.521+0000 7fc7982b7700 1 -- 172.21.15.149:0/883257348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7900f5410 msgr2=0x7fc7900fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:46.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.521+0000 7fc7982b7700 1 --2- 172.21.15.149:0/883257348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7900f5410 0x7fc7900fd4e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fc780005010 tx=0x7fc78000b040).stop 2024-02-12T16:34:46.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.521+0000 7fc7982b7700 1 -- 172.21.15.149:0/883257348 shutdown_connections 2024-02-12T16:34:46.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.521+0000 7fc7982b7700 1 --2- 172.21.15.149:0/883257348 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc77c02ff10 0x7fc77c0323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:46.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.521+0000 7fc7982b7700 1 --2- 172.21.15.149:0/883257348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7900f5410 0x7fc7900fd4e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:46.523 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.521+0000 7fc7982b7700 1 -- 172.21.15.149:0/883257348 >> 172.21.15.149:0/883257348 conn(0x7fc7900f0b70 msgr2=0x7fc79005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:46.524 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.521+0000 7fc7982b7700 1 -- 172.21.15.149:0/883257348 shutdown_connections 2024-02-12T16:34:46.524 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:46.521+0000 7fc7982b7700 1 -- 172.21.15.149:0/883257348 wait complete. 2024-02-12T16:34:46.524 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:47.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:47 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:46.520393+0000 mon.smithi104 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.149:0/883257348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:48.149 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:48.150 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:48.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:46.322393+0000 mgr.smithi104.ohwbhf (mgr.14162) 162 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:49.985 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.982+0000 7f30c8c85700 1 -- 172.21.15.149:0/2644693336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c40f6510 msgr2=0x7f30c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:49.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.982+0000 7f30c8c85700 1 --2- 172.21.15.149:0/2644693336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c40f6510 0x7f30c40f6930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f30b8009fb0 tx=0x7f30b8009b30).stop 2024-02-12T16:34:49.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.983+0000 7f30c8c85700 1 -- 172.21.15.149:0/2644693336 shutdown_connections 2024-02-12T16:34:49.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.983+0000 7f30c8c85700 1 --2- 172.21.15.149:0/2644693336 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c40f6510 0x7f30c40f6930 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:49.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.983+0000 7f30c8c85700 1 -- 172.21.15.149:0/2644693336 >> 172.21.15.149:0/2644693336 conn(0x7f30c40f1c70 msgr2=0x7f30c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:49.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.983+0000 7f30c8c85700 1 -- 172.21.15.149:0/2644693336 shutdown_connections 2024-02-12T16:34:49.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.983+0000 7f30c8c85700 1 -- 172.21.15.149:0/2644693336 wait complete. 2024-02-12T16:34:49.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.983+0000 7f30c8c85700 1 Processor -- start 2024-02-12T16:34:49.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.983+0000 7f30c8c85700 1 -- start start 2024-02-12T16:34:49.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30c8c85700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c40f6510 0x7f30c41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:49.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30c8c85700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4106cd0 con 0x7f30c40f6510 2024-02-12T16:34:49.988 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30c259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c40f6510 0x7f30c41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:49.988 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30c259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c40f6510 0x7f30c41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33198/0 (socket says 172.21.15.149:33198) 2024-02-12T16:34:49.988 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30c259c700 1 -- 172.21.15.149:0/4222167641 learned_addr learned my addr 172.21.15.149:0/4222167641 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:49.988 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30c259c700 1 -- 172.21.15.149:0/4222167641 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b8005040 con 0x7f30c40f6510 2024-02-12T16:34:49.988 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30c259c700 1 --2- 172.21.15.149:0/4222167641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c40f6510 0x7f30c41067e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f30b80089f0 tx=0x7f30b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:49.988 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30b2ffd700 1 -- 172.21.15.149:0/4222167641 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30b8009320 con 0x7f30c40f6510 2024-02-12T16:34:49.988 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30c8c85700 1 -- 172.21.15.149:0/4222167641 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c4106ed0 con 0x7f30c40f6510 2024-02-12T16:34:49.989 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30b2ffd700 1 -- 172.21.15.149:0/4222167641 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30b8013380 con 0x7f30c40f6510 2024-02-12T16:34:49.989 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30b2ffd700 1 -- 172.21.15.149:0/4222167641 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30b801ed80 con 0x7f30c40f6510 2024-02-12T16:34:49.989 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.984+0000 7f30c8c85700 1 -- 172.21.15.149:0/4222167641 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c4107360 con 0x7f30c40f6510 2024-02-12T16:34:49.989 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.985+0000 7f30b2ffd700 1 -- 172.21.15.149:0/4222167641 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30b801e420 con 0x7f30c40f6510 2024-02-12T16:34:49.989 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.985+0000 7f30c8c85700 1 -- 172.21.15.149:0/4222167641 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c4045ad0 con 0x7f30c40f6510 2024-02-12T16:34:49.989 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.985+0000 7f30b2ffd700 1 --2- 172.21.15.149:0/4222167641 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f30ac02ff60 0x7f30ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:49.989 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.985+0000 7f30b2ffd700 1 -- 172.21.15.149:0/4222167641 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30b8041120 con 0x7f30c40f6510 2024-02-12T16:34:49.989 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.986+0000 7f30c1d9b700 1 --2- 172.21.15.149:0/4222167641 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f30ac02ff60 0x7f30ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:49.989 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.986+0000 7f30c1d9b700 1 --2- 172.21.15.149:0/4222167641 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f30ac02ff60 0x7f30ac032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f30b40096a0 tx=0x7f30b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:49.992 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:49.990+0000 7f30b2ffd700 1 -- 172.21.15.149:0/4222167641 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30b80095f0 con 0x7f30c40f6510 2024-02-12T16:34:50.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.196+0000 7f30c8c85700 1 -- 172.21.15.149:0/4222167641 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30c41081b0 con 0x7f30c40f6510 2024-02-12T16:34:50.199 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.197+0000 7f30b2ffd700 1 -- 172.21.15.149:0/4222167641 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f30b801cb10 con 0x7f30c40f6510 2024-02-12T16:34:50.199 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:50.199 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:50.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.198+0000 7f30c8c85700 1 -- 172.21.15.149:0/4222167641 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f30ac02ff60 msgr2=0x7f30ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:50.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.198+0000 7f30c8c85700 1 --2- 172.21.15.149:0/4222167641 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f30ac02ff60 0x7f30ac032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f30b40096a0 tx=0x7f30b4006b40).stop 2024-02-12T16:34:50.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.198+0000 7f30c8c85700 1 -- 172.21.15.149:0/4222167641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c40f6510 msgr2=0x7f30c41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:50.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.198+0000 7f30c8c85700 1 --2- 172.21.15.149:0/4222167641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c40f6510 0x7f30c41067e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f30b80089f0 tx=0x7f30b800b040).stop 2024-02-12T16:34:50.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.198+0000 7f30c8c85700 1 -- 172.21.15.149:0/4222167641 shutdown_connections 2024-02-12T16:34:50.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.198+0000 7f30c8c85700 1 --2- 172.21.15.149:0/4222167641 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f30ac02ff60 0x7f30ac032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:50.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.198+0000 7f30c8c85700 1 --2- 172.21.15.149:0/4222167641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30c40f6510 0x7f30c41067e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:50.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.198+0000 7f30c8c85700 1 -- 172.21.15.149:0/4222167641 >> 172.21.15.149:0/4222167641 conn(0x7f30c40f1c70 msgr2=0x7f30c40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:50.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.198+0000 7f30c8c85700 1 -- 172.21.15.149:0/4222167641 shutdown_connections 2024-02-12T16:34:50.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:50.198+0000 7f30c8c85700 1 -- 172.21.15.149:0/4222167641 wait complete. 2024-02-12T16:34:50.202 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:50.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:48.322770+0000 mgr.smithi104.ohwbhf (mgr.14162) 163 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:51.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:51 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:50.198088+0000 mon.smithi104 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.149:0/4222167641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:51.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:51.823 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:52.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:50.323024+0000 mgr.smithi104.ohwbhf (mgr.14162) 164 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:53.589 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.585+0000 7fb002d42700 1 -- 172.21.15.149:0/3807833015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faffc0f5410 msgr2=0x7faffc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:53.589 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.585+0000 7fb002d42700 1 --2- 172.21.15.149:0/3807833015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faffc0f5410 0x7faffc0f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7faff0009fb0 tx=0x7faff0009b30).stop 2024-02-12T16:34:53.589 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.585+0000 7fb002d42700 1 -- 172.21.15.149:0/3807833015 shutdown_connections 2024-02-12T16:34:53.589 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.585+0000 7fb002d42700 1 --2- 172.21.15.149:0/3807833015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faffc0f5410 0x7faffc0f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:53.589 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.585+0000 7fb002d42700 1 -- 172.21.15.149:0/3807833015 >> 172.21.15.149:0/3807833015 conn(0x7faffc0f0b70 msgr2=0x7faffc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:53.589 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.585+0000 7fb002d42700 1 -- 172.21.15.149:0/3807833015 shutdown_connections 2024-02-12T16:34:53.589 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.585+0000 7fb002d42700 1 -- 172.21.15.149:0/3807833015 wait complete. 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.586+0000 7fb002d42700 1 Processor -- start 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.586+0000 7fb002d42700 1 -- start start 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.586+0000 7fb002d42700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faffc0f5410 0x7faffc0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.586+0000 7fb002d42700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc0fb960 con 0x7faffc0f5410 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.586+0000 7fb000ade700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faffc0f5410 0x7faffc0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.586+0000 7fb000ade700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faffc0f5410 0x7faffc0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33212/0 (socket says 172.21.15.149:33212) 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.586+0000 7fb000ade700 1 -- 172.21.15.149:0/708883092 learned_addr learned my addr 172.21.15.149:0/708883092 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.587+0000 7fb000ade700 1 -- 172.21.15.149:0/708883092 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff0005040 con 0x7faffc0f5410 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.587+0000 7fb000ade700 1 --2- 172.21.15.149:0/708883092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faffc0f5410 0x7faffc0fd4e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7faff0005010 tx=0x7faff000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.587+0000 7faff97fa700 1 -- 172.21.15.149:0/708883092 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faff0009320 con 0x7faffc0f5410 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.587+0000 7faff97fa700 1 -- 172.21.15.149:0/708883092 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faff0013380 con 0x7faffc0f5410 2024-02-12T16:34:53.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.587+0000 7fb002d42700 1 -- 172.21.15.149:0/708883092 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc0fbb00 con 0x7faffc0f5410 2024-02-12T16:34:53.591 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.587+0000 7faff97fa700 1 -- 172.21.15.149:0/708883092 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faff0014e50 con 0x7faffc0f5410 2024-02-12T16:34:53.591 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.587+0000 7fb002d42700 1 -- 172.21.15.149:0/708883092 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc0fbf90 con 0x7faffc0f5410 2024-02-12T16:34:53.591 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.588+0000 7faff97fa700 1 -- 172.21.15.149:0/708883092 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faff0014420 con 0x7faffc0f5410 2024-02-12T16:34:53.592 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.588+0000 7fb002d42700 1 -- 172.21.15.149:0/708883092 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc059100 con 0x7faffc0f5410 2024-02-12T16:34:53.592 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.588+0000 7faff97fa700 1 --2- 172.21.15.149:0/708883092 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafec02ff10 0x7fafec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:53.592 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.588+0000 7faff97fa700 1 -- 172.21.15.149:0/708883092 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faff0040f70 con 0x7faffc0f5410 2024-02-12T16:34:53.592 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.588+0000 7faffbfff700 1 --2- 172.21.15.149:0/708883092 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafec02ff10 0x7fafec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:53.592 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.588+0000 7faffbfff700 1 --2- 172.21.15.149:0/708883092 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafec02ff10 0x7fafec0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fafe80096a0 tx=0x7fafe8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:53.595 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.593+0000 7faff97fa700 1 -- 172.21.15.149:0/708883092 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faff001dd50 con 0x7faffc0f5410 2024-02-12T16:34:53.798 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.796+0000 7fb002d42700 1 -- 172.21.15.149:0/708883092 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faffc0fafa0 con 0x7faffc0f5410 2024-02-12T16:34:53.798 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.796+0000 7faff97fa700 1 -- 172.21.15.149:0/708883092 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faff0012330 con 0x7faffc0f5410 2024-02-12T16:34:53.799 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:53.799 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:53.800 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.798+0000 7fb002d42700 1 -- 172.21.15.149:0/708883092 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafec02ff10 msgr2=0x7fafec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:53.800 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.798+0000 7fb002d42700 1 --2- 172.21.15.149:0/708883092 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafec02ff10 0x7fafec0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fafe80096a0 tx=0x7fafe8006b40).stop 2024-02-12T16:34:53.800 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.798+0000 7fb002d42700 1 -- 172.21.15.149:0/708883092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faffc0f5410 msgr2=0x7faffc0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:53.800 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.798+0000 7fb002d42700 1 --2- 172.21.15.149:0/708883092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faffc0f5410 0x7faffc0fd4e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7faff0005010 tx=0x7faff000b040).stop 2024-02-12T16:34:53.801 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.798+0000 7fb002d42700 1 -- 172.21.15.149:0/708883092 shutdown_connections 2024-02-12T16:34:53.801 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.798+0000 7fb002d42700 1 --2- 172.21.15.149:0/708883092 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafec02ff10 0x7fafec0323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:53.801 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.798+0000 7fb002d42700 1 --2- 172.21.15.149:0/708883092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faffc0f5410 0x7faffc0fd4e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:53.801 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.798+0000 7fb002d42700 1 -- 172.21.15.149:0/708883092 >> 172.21.15.149:0/708883092 conn(0x7faffc0f0b70 msgr2=0x7faffc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:53.801 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.798+0000 7fb002d42700 1 -- 172.21.15.149:0/708883092 shutdown_connections 2024-02-12T16:34:53.801 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:53.798+0000 7fb002d42700 1 -- 172.21.15.149:0/708883092 wait complete. 2024-02-12T16:34:53.802 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:54.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:52.323347+0000 mgr.smithi104.ohwbhf (mgr.14162) 165 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:54.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:53.797628+0000 mon.smithi104 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.149:0/708883092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:55.484 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:55.484 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:34:56.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:54.323732+0000 mgr.smithi104.ohwbhf (mgr.14162) 166 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:57.357 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.355+0000 7fe173fff700 1 -- 172.21.15.149:0/4071355243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1740f51a0 msgr2=0x7fe1740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:57.357 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.355+0000 7fe173fff700 1 --2- 172.21.15.149:0/4071355243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1740f51a0 0x7fe1740f55c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fe164009fb0 tx=0x7fe164009b30).stop 2024-02-12T16:34:57.357 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.355+0000 7fe173fff700 1 -- 172.21.15.149:0/4071355243 shutdown_connections 2024-02-12T16:34:57.357 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.355+0000 7fe173fff700 1 --2- 172.21.15.149:0/4071355243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1740f51a0 0x7fe1740f55c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:57.357 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.355+0000 7fe173fff700 1 -- 172.21.15.149:0/4071355243 >> 172.21.15.149:0/4071355243 conn(0x7fe1740f09a0 msgr2=0x7fe1740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:57.357 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.355+0000 7fe173fff700 1 -- 172.21.15.149:0/4071355243 shutdown_connections 2024-02-12T16:34:57.358 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.355+0000 7fe173fff700 1 -- 172.21.15.149:0/4071355243 wait complete. 2024-02-12T16:34:57.359 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.355+0000 7fe173fff700 1 Processor -- start 2024-02-12T16:34:57.359 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.356+0000 7fe173fff700 1 -- start start 2024-02-12T16:34:57.359 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.356+0000 7fe173fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1740f51a0 0x7fe174109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:57.359 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.356+0000 7fe173fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe174060280 con 0x7fe1740f51a0 2024-02-12T16:34:57.359 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.356+0000 7fe172ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1740f51a0 0x7fe174109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:57.359 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.356+0000 7fe172ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1740f51a0 0x7fe174109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:43312/0 (socket says 172.21.15.149:43312) 2024-02-12T16:34:57.359 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.356+0000 7fe172ffd700 1 -- 172.21.15.149:0/2521364113 learned_addr learned my addr 172.21.15.149:0/2521364113 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:34:57.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.356+0000 7fe172ffd700 1 -- 172.21.15.149:0/2521364113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe164005040 con 0x7fe1740f51a0 2024-02-12T16:34:57.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.357+0000 7fe172ffd700 1 --2- 172.21.15.149:0/2521364113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1740f51a0 0x7fe174109290 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fe1640089f0 tx=0x7fe16400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:57.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.357+0000 7fe15b7fe700 1 -- 172.21.15.149:0/2521364113 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe164009320 con 0x7fe1740f51a0 2024-02-12T16:34:57.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.357+0000 7fe173fff700 1 -- 172.21.15.149:0/2521364113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe174060480 con 0x7fe1740f51a0 2024-02-12T16:34:57.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.357+0000 7fe15b7fe700 1 -- 172.21.15.149:0/2521364113 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe164013380 con 0x7fe1740f51a0 2024-02-12T16:34:57.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.357+0000 7fe15b7fe700 1 -- 172.21.15.149:0/2521364113 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe164014e20 con 0x7fe1740f51a0 2024-02-12T16:34:57.360 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.357+0000 7fe173fff700 1 -- 172.21.15.149:0/2521364113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1741099c0 con 0x7fe1740f51a0 2024-02-12T16:34:57.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.358+0000 7fe15b7fe700 1 -- 172.21.15.149:0/2521364113 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe164014420 con 0x7fe1740f51a0 2024-02-12T16:34:57.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.358+0000 7fe173fff700 1 -- 172.21.15.149:0/2521364113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe174045ad0 con 0x7fe1740f51a0 2024-02-12T16:34:57.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.358+0000 7fe15b7fe700 1 --2- 172.21.15.149:0/2521364113 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 0x7fe15c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:34:57.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.358+0000 7fe15b7fe700 1 -- 172.21.15.149:0/2521364113 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe164040ed0 con 0x7fe1740f51a0 2024-02-12T16:34:57.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.358+0000 7fe1727fc700 1 --2- 172.21.15.149:0/2521364113 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 0x7fe15c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:34:57.361 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.358+0000 7fe1727fc700 1 --2- 172.21.15.149:0/2521364113 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 0x7fe15c0323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fe1680096a0 tx=0x7fe168006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:34:57.364 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.362+0000 7fe15b7fe700 1 -- 172.21.15.149:0/2521364113 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe164014790 con 0x7fe1740f51a0 2024-02-12T16:34:57.569 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.567+0000 7fe173fff700 1 -- 172.21.15.149:0/2521364113 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1740fad30 con 0x7fe1740f51a0 2024-02-12T16:34:57.569 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.567+0000 7fe15b7fe700 1 -- 172.21.15.149:0/2521364113 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe16401d7f0 con 0x7fe1740f51a0 2024-02-12T16:34:57.570 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:34:57.570 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:34:57.571 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.569+0000 7fe173fff700 1 -- 172.21.15.149:0/2521364113 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 msgr2=0x7fe15c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:57.571 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.569+0000 7fe173fff700 1 --2- 172.21.15.149:0/2521364113 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 0x7fe15c0323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fe1680096a0 tx=0x7fe168006b40).stop 2024-02-12T16:34:57.571 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.569+0000 7fe173fff700 1 -- 172.21.15.149:0/2521364113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1740f51a0 msgr2=0x7fe174109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:34:57.571 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.569+0000 7fe173fff700 1 --2- 172.21.15.149:0/2521364113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1740f51a0 0x7fe174109290 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fe1640089f0 tx=0x7fe16400b040).stop 2024-02-12T16:34:57.571 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.569+0000 7fe173fff700 1 -- 172.21.15.149:0/2521364113 shutdown_connections 2024-02-12T16:34:57.572 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.569+0000 7fe173fff700 1 --2- 172.21.15.149:0/2521364113 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe15c02ff10 0x7fe15c0323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:57.572 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.569+0000 7fe173fff700 1 --2- 172.21.15.149:0/2521364113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1740f51a0 0x7fe174109290 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:34:57.572 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.569+0000 7fe173fff700 1 -- 172.21.15.149:0/2521364113 >> 172.21.15.149:0/2521364113 conn(0x7fe1740f09a0 msgr2=0x7fe1740f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:34:57.572 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.569+0000 7fe173fff700 1 -- 172.21.15.149:0/2521364113 shutdown_connections 2024-02-12T16:34:57.572 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:34:57.569+0000 7fe173fff700 1 -- 172.21.15.149:0/2521364113 wait complete. 2024-02-12T16:34:57.572 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:34:58.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:56.324009+0000 mgr.smithi104.ohwbhf (mgr.14162) 167 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:34:58.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:34:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:34:57.568617+0000 mon.smithi104 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.149:0/2521364113' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:34:59.189 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:34:59.189 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:00.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:34:58.324365+0000 mgr.smithi104.ohwbhf (mgr.14162) 168 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:01.046 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.042+0000 7f75dd52f700 1 -- 172.21.15.149:0/3575148063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d80f51a0 msgr2=0x7f75d80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:01.046 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.042+0000 7f75dd52f700 1 --2- 172.21.15.149:0/3575148063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d80f51a0 0x7f75d80f55c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f75c8009fb0 tx=0x7f75c8009b30).stop 2024-02-12T16:35:01.046 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.042+0000 7f75dd52f700 1 -- 172.21.15.149:0/3575148063 shutdown_connections 2024-02-12T16:35:01.046 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.042+0000 7f75dd52f700 1 --2- 172.21.15.149:0/3575148063 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d80f51a0 0x7f75d80f55c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:01.046 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.042+0000 7f75dd52f700 1 -- 172.21.15.149:0/3575148063 >> 172.21.15.149:0/3575148063 conn(0x7f75d80f09a0 msgr2=0x7f75d80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.042+0000 7f75dd52f700 1 -- 172.21.15.149:0/3575148063 shutdown_connections 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.042+0000 7f75dd52f700 1 -- 172.21.15.149:0/3575148063 wait complete. 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.043+0000 7f75dd52f700 1 Processor -- start 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.043+0000 7f75dd52f700 1 -- start start 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.043+0000 7f75dd52f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d80f51a0 0x7f75d81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.043+0000 7f75dd52f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d8106ac0 con 0x7f75d80f51a0 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.043+0000 7f75d7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d80f51a0 0x7f75d81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.043+0000 7f75d7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d80f51a0 0x7f75d81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:43324/0 (socket says 172.21.15.149:43324) 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.043+0000 7f75d7fff700 1 -- 172.21.15.149:0/3182593812 learned_addr learned my addr 172.21.15.149:0/3182593812 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.044+0000 7f75d7fff700 1 -- 172.21.15.149:0/3182593812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c8005040 con 0x7f75d80f51a0 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.044+0000 7f75d7fff700 1 --2- 172.21.15.149:0/3182593812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d80f51a0 0x7f75d81065d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f75c80089f0 tx=0x7f75c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.044+0000 7f75d4ff9700 1 -- 172.21.15.149:0/3182593812 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c8009320 con 0x7f75d80f51a0 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.044+0000 7f75dd52f700 1 -- 172.21.15.149:0/3182593812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d8106cc0 con 0x7f75d80f51a0 2024-02-12T16:35:01.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.044+0000 7f75d4ff9700 1 -- 172.21.15.149:0/3182593812 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75c8013380 con 0x7f75d80f51a0 2024-02-12T16:35:01.048 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.044+0000 7f75d4ff9700 1 -- 172.21.15.149:0/3182593812 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75c8014e20 con 0x7f75d80f51a0 2024-02-12T16:35:01.048 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.044+0000 7f75dd52f700 1 -- 172.21.15.149:0/3182593812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d8107150 con 0x7f75d80f51a0 2024-02-12T16:35:01.048 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.045+0000 7f75d4ff9700 1 -- 172.21.15.149:0/3182593812 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75c8014420 con 0x7f75d80f51a0 2024-02-12T16:35:01.049 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.045+0000 7f75d4ff9700 1 --2- 172.21.15.149:0/3182593812 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff10 0x7f75c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:01.049 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.045+0000 7f75dd52f700 1 -- 172.21.15.149:0/3182593812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d8045ad0 con 0x7f75d80f51a0 2024-02-12T16:35:01.049 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.045+0000 7f75d4ff9700 1 -- 172.21.15.149:0/3182593812 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75c8040fd0 con 0x7f75d80f51a0 2024-02-12T16:35:01.049 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.045+0000 7f75d77fe700 1 --2- 172.21.15.149:0/3182593812 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff10 0x7f75c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:01.049 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.045+0000 7f75d77fe700 1 --2- 172.21.15.149:0/3182593812 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff10 0x7f75c00323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f75cc0096a0 tx=0x7f75cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:01.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.050+0000 7f75d4ff9700 1 -- 172.21.15.149:0/3182593812 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75c801dd50 con 0x7f75d80f51a0 2024-02-12T16:35:01.257 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.255+0000 7f75dd52f700 1 -- 172.21.15.149:0/3182593812 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75d80fad30 con 0x7f75d80f51a0 2024-02-12T16:35:01.257 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.256+0000 7f75d4ff9700 1 -- 172.21.15.149:0/3182593812 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f75c801db70 con 0x7f75d80f51a0 2024-02-12T16:35:01.258 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:01.258 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:01.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.257+0000 7f75dd52f700 1 -- 172.21.15.149:0/3182593812 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff10 msgr2=0x7f75c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:01.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.257+0000 7f75dd52f700 1 --2- 172.21.15.149:0/3182593812 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff10 0x7f75c00323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f75cc0096a0 tx=0x7f75cc006b40).stop 2024-02-12T16:35:01.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.257+0000 7f75dd52f700 1 -- 172.21.15.149:0/3182593812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d80f51a0 msgr2=0x7f75d81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:01.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.257+0000 7f75dd52f700 1 --2- 172.21.15.149:0/3182593812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d80f51a0 0x7f75d81065d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f75c80089f0 tx=0x7f75c800b040).stop 2024-02-12T16:35:01.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.257+0000 7f75dd52f700 1 -- 172.21.15.149:0/3182593812 shutdown_connections 2024-02-12T16:35:01.260 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.257+0000 7f75dd52f700 1 --2- 172.21.15.149:0/3182593812 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f75c002ff10 0x7f75c00323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:01.260 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.257+0000 7f75dd52f700 1 --2- 172.21.15.149:0/3182593812 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d80f51a0 0x7f75d81065d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:01.260 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.257+0000 7f75dd52f700 1 -- 172.21.15.149:0/3182593812 >> 172.21.15.149:0/3182593812 conn(0x7f75d80f09a0 msgr2=0x7f75d805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:01.260 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.257+0000 7f75dd52f700 1 -- 172.21.15.149:0/3182593812 shutdown_connections 2024-02-12T16:35:01.260 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:01.257+0000 7f75dd52f700 1 -- 172.21.15.149:0/3182593812 wait complete. 2024-02-12T16:35:01.260 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:02.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:00.324621+0000 mgr.smithi104.ohwbhf (mgr.14162) 169 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:02.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:01.256902+0000 mon.smithi104 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.149:0/3182593812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:02.912 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:02.912 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:04.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:02.324945+0000 mgr.smithi104.ohwbhf (mgr.14162) 170 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:04.732 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.728+0000 7f3e28387700 1 -- 172.21.15.149:0/1012322694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e200f6510 msgr2=0x7f3e200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:04.732 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.728+0000 7f3e28387700 1 --2- 172.21.15.149:0/1012322694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e200f6510 0x7f3e200f6930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f3e1c009fb0 tx=0x7f3e1c009b30).stop 2024-02-12T16:35:04.732 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.728+0000 7f3e28387700 1 -- 172.21.15.149:0/1012322694 shutdown_connections 2024-02-12T16:35:04.732 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.728+0000 7f3e28387700 1 --2- 172.21.15.149:0/1012322694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e200f6510 0x7f3e200f6930 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:04.732 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.728+0000 7f3e28387700 1 -- 172.21.15.149:0/1012322694 >> 172.21.15.149:0/1012322694 conn(0x7f3e200f1c70 msgr2=0x7f3e200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:04.732 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.728+0000 7f3e28387700 1 -- 172.21.15.149:0/1012322694 shutdown_connections 2024-02-12T16:35:04.732 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.728+0000 7f3e28387700 1 -- 172.21.15.149:0/1012322694 wait complete. 2024-02-12T16:35:04.732 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.729+0000 7f3e28387700 1 Processor -- start 2024-02-12T16:35:04.732 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.729+0000 7f3e28387700 1 -- start start 2024-02-12T16:35:04.732 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.729+0000 7f3e28387700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e200f6510 0x7f3e201067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:04.733 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.729+0000 7f3e28387700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e20106cd0 con 0x7f3e200f6510 2024-02-12T16:35:04.733 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.729+0000 7f3e26123700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e200f6510 0x7f3e201067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:04.733 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.729+0000 7f3e26123700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e200f6510 0x7f3e201067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:39892/0 (socket says 172.21.15.149:39892) 2024-02-12T16:35:04.733 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.729+0000 7f3e26123700 1 -- 172.21.15.149:0/3836993907 learned_addr learned my addr 172.21.15.149:0/3836993907 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:04.733 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.729+0000 7f3e26123700 1 -- 172.21.15.149:0/3836993907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1c005040 con 0x7f3e200f6510 2024-02-12T16:35:04.733 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.729+0000 7f3e26123700 1 --2- 172.21.15.149:0/3836993907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e200f6510 0x7f3e201067e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f3e1c005010 tx=0x7f3e1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:04.733 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.730+0000 7f3e16ffd700 1 -- 172.21.15.149:0/3836993907 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e1c009320 con 0x7f3e200f6510 2024-02-12T16:35:04.733 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.730+0000 7f3e28387700 1 -- 172.21.15.149:0/3836993907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e20106ed0 con 0x7f3e200f6510 2024-02-12T16:35:04.733 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.730+0000 7f3e16ffd700 1 -- 172.21.15.149:0/3836993907 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e1c013380 con 0x7f3e200f6510 2024-02-12T16:35:04.733 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.730+0000 7f3e16ffd700 1 -- 172.21.15.149:0/3836993907 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e1c014e50 con 0x7f3e200f6510 2024-02-12T16:35:04.733 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.730+0000 7f3e28387700 1 -- 172.21.15.149:0/3836993907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e20107360 con 0x7f3e200f6510 2024-02-12T16:35:04.734 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.730+0000 7f3e16ffd700 1 -- 172.21.15.149:0/3836993907 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e1c0095f0 con 0x7f3e200f6510 2024-02-12T16:35:04.734 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.731+0000 7f3e16ffd700 1 --2- 172.21.15.149:0/3836993907 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3e0c02ff10 0x7f3e0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:04.734 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.731+0000 7f3e28387700 1 -- 172.21.15.149:0/3836993907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e20045ad0 con 0x7f3e200f6510 2024-02-12T16:35:04.734 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.731+0000 7f3e16ffd700 1 -- 172.21.15.149:0/3836993907 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e1c040f10 con 0x7f3e200f6510 2024-02-12T16:35:04.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.731+0000 7f3e25922700 1 --2- 172.21.15.149:0/3836993907 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3e0c02ff10 0x7f3e0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:04.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.731+0000 7f3e25922700 1 --2- 172.21.15.149:0/3836993907 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3e0c02ff10 0x7f3e0c0323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3e100096a0 tx=0x7f3e10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:04.737 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.735+0000 7f3e16ffd700 1 -- 172.21.15.149:0/3836993907 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e1c01dbc0 con 0x7f3e200f6510 2024-02-12T16:35:04.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.939+0000 7f3e28387700 1 -- 172.21.15.149:0/3836993907 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e20108210 con 0x7f3e200f6510 2024-02-12T16:35:04.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.940+0000 7f3e16ffd700 1 -- 172.21.15.149:0/3836993907 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3e1c010230 con 0x7f3e200f6510 2024-02-12T16:35:04.942 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:04.942 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:04.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.941+0000 7f3e28387700 1 -- 172.21.15.149:0/3836993907 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3e0c02ff10 msgr2=0x7f3e0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:04.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.941+0000 7f3e28387700 1 --2- 172.21.15.149:0/3836993907 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3e0c02ff10 0x7f3e0c0323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3e100096a0 tx=0x7f3e10006b40).stop 2024-02-12T16:35:04.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.941+0000 7f3e28387700 1 -- 172.21.15.149:0/3836993907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e200f6510 msgr2=0x7f3e201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:04.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.941+0000 7f3e28387700 1 --2- 172.21.15.149:0/3836993907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e200f6510 0x7f3e201067e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f3e1c005010 tx=0x7f3e1c00b040).stop 2024-02-12T16:35:04.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.941+0000 7f3e28387700 1 -- 172.21.15.149:0/3836993907 shutdown_connections 2024-02-12T16:35:04.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.941+0000 7f3e28387700 1 --2- 172.21.15.149:0/3836993907 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3e0c02ff10 0x7f3e0c0323d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:04.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.941+0000 7f3e28387700 1 --2- 172.21.15.149:0/3836993907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e200f6510 0x7f3e201067e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:04.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.941+0000 7f3e28387700 1 -- 172.21.15.149:0/3836993907 >> 172.21.15.149:0/3836993907 conn(0x7f3e200f1c70 msgr2=0x7f3e200f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:04.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.941+0000 7f3e28387700 1 -- 172.21.15.149:0/3836993907 shutdown_connections 2024-02-12T16:35:04.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:04.941+0000 7f3e28387700 1 -- 172.21.15.149:0/3836993907 wait complete. 2024-02-12T16:35:04.944 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:05.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:04.941001+0000 mon.smithi104 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.149:0/3836993907' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:06.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:04.325184+0000 mgr.smithi104.ohwbhf (mgr.14162) 171 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:06.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:06.597 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:08.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.410+0000 7f9b599e6700 1 -- 172.21.15.149:0/3604831889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b540f4860 msgr2=0x7f9b540f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:08.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.410+0000 7f9b599e6700 1 --2- 172.21.15.149:0/3604831889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b540f4860 0x7f9b540f4c80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f9b48009fb0 tx=0x7f9b48009b30).stop 2024-02-12T16:35:08.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.410+0000 7f9b599e6700 1 -- 172.21.15.149:0/3604831889 shutdown_connections 2024-02-12T16:35:08.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.410+0000 7f9b599e6700 1 --2- 172.21.15.149:0/3604831889 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b540f4860 0x7f9b540f4c80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:08.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.410+0000 7f9b599e6700 1 -- 172.21.15.149:0/3604831889 >> 172.21.15.149:0/3604831889 conn(0x7f9b540f02e0 msgr2=0x7f9b540f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:08.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.410+0000 7f9b599e6700 1 -- 172.21.15.149:0/3604831889 shutdown_connections 2024-02-12T16:35:08.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.410+0000 7f9b599e6700 1 -- 172.21.15.149:0/3604831889 wait complete. 2024-02-12T16:35:08.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.411+0000 7f9b599e6700 1 Processor -- start 2024-02-12T16:35:08.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.411+0000 7f9b599e6700 1 -- start start 2024-02-12T16:35:08.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.411+0000 7f9b599e6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b540f4860 0x7f9b540f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:08.414 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.411+0000 7f9b599e6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b540f58d0 con 0x7f9b540f4860 2024-02-12T16:35:08.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.411+0000 7f9b52ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b540f4860 0x7f9b540f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:08.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.411+0000 7f9b52ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b540f4860 0x7f9b540f7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:39924/0 (socket says 172.21.15.149:39924) 2024-02-12T16:35:08.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.411+0000 7f9b52ffd700 1 -- 172.21.15.149:0/1888736325 learned_addr learned my addr 172.21.15.149:0/1888736325 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:08.415 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.412+0000 7f9b52ffd700 1 -- 172.21.15.149:0/1888736325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b48005040 con 0x7f9b540f4860 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.412+0000 7f9b52ffd700 1 --2- 172.21.15.149:0/1888736325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b540f4860 0x7f9b540f7450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9b480089f0 tx=0x7f9b4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.412+0000 7f9b3b7fe700 1 -- 172.21.15.149:0/1888736325 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b48009320 con 0x7f9b540f4860 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.412+0000 7f9b599e6700 1 -- 172.21.15.149:0/1888736325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b540f5a70 con 0x7f9b540f4860 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.412+0000 7f9b3b7fe700 1 -- 172.21.15.149:0/1888736325 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b48013380 con 0x7f9b540f4860 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.412+0000 7f9b3b7fe700 1 -- 172.21.15.149:0/1888736325 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b48014e20 con 0x7f9b540f4860 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.412+0000 7f9b599e6700 1 -- 172.21.15.149:0/1888736325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b540f5f00 con 0x7f9b540f4860 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.413+0000 7f9b3b7fe700 1 -- 172.21.15.149:0/1888736325 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b480263a0 con 0x7f9b540f4860 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.413+0000 7f9b599e6700 1 -- 172.21.15.149:0/1888736325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b54059100 con 0x7f9b540f4860 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.413+0000 7f9b3b7fe700 1 --2- 172.21.15.149:0/1888736325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9b3c02ff10 0x7f9b3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.413+0000 7f9b3b7fe700 1 -- 172.21.15.149:0/1888736325 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b480412c0 con 0x7f9b540f4860 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.413+0000 7f9b527fc700 1 --2- 172.21.15.149:0/1888736325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9b3c02ff10 0x7f9b3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:08.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.413+0000 7f9b527fc700 1 --2- 172.21.15.149:0/1888736325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9b3c02ff10 0x7f9b3c0323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f9b440096a0 tx=0x7f9b44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:08.419 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.418+0000 7f9b3b7fe700 1 -- 172.21.15.149:0/1888736325 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b480126f0 con 0x7f9b540f4860 2024-02-12T16:35:08.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:06.325504+0000 mgr.smithi104.ohwbhf (mgr.14162) 172 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:08.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.621+0000 7f9b599e6700 1 -- 172.21.15.149:0/1888736325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b54045ad0 con 0x7f9b540f4860 2024-02-12T16:35:08.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.622+0000 7f9b3b7fe700 1 -- 172.21.15.149:0/1888736325 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9b480126f0 con 0x7f9b540f4860 2024-02-12T16:35:08.624 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:08.624 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:08.625 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.623+0000 7f9b599e6700 1 -- 172.21.15.149:0/1888736325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9b3c02ff10 msgr2=0x7f9b3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:08.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.623+0000 7f9b599e6700 1 --2- 172.21.15.149:0/1888736325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9b3c02ff10 0x7f9b3c0323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f9b440096a0 tx=0x7f9b44006b40).stop 2024-02-12T16:35:08.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.623+0000 7f9b599e6700 1 -- 172.21.15.149:0/1888736325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b540f4860 msgr2=0x7f9b540f7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:08.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.623+0000 7f9b599e6700 1 --2- 172.21.15.149:0/1888736325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b540f4860 0x7f9b540f7450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9b480089f0 tx=0x7f9b4800b040).stop 2024-02-12T16:35:08.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.623+0000 7f9b599e6700 1 -- 172.21.15.149:0/1888736325 shutdown_connections 2024-02-12T16:35:08.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.623+0000 7f9b599e6700 1 --2- 172.21.15.149:0/1888736325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9b3c02ff10 0x7f9b3c0323d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:08.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.623+0000 7f9b599e6700 1 --2- 172.21.15.149:0/1888736325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b540f4860 0x7f9b540f7450 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:08.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.623+0000 7f9b599e6700 1 -- 172.21.15.149:0/1888736325 >> 172.21.15.149:0/1888736325 conn(0x7f9b540f02e0 msgr2=0x7f9b540f0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:08.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.623+0000 7f9b599e6700 1 -- 172.21.15.149:0/1888736325 shutdown_connections 2024-02-12T16:35:08.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:08.623+0000 7f9b599e6700 1 -- 172.21.15.149:0/1888736325 wait complete. 2024-02-12T16:35:08.627 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:09.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:08.622904+0000 mon.smithi104 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.149:0/1888736325' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:10.305 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:10.306 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:10.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:08.325844+0000 mgr.smithi104.ohwbhf (mgr.14162) 173 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:11.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:10.436716+0000 mon.smithi104 (mon.0) 211 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-12T16:35:11.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:10.442015+0000 mon.smithi104 (mon.0) 212 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-12T16:35:11.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:10.442812+0000 mon.smithi104 (mon.0) 213 : audit [DBG] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:35:12.118 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.116+0000 7f767754d700 1 -- 172.21.15.149:0/67855816 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76700f5410 msgr2=0x7f76700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:12.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.116+0000 7f767754d700 1 --2- 172.21.15.149:0/67855816 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76700f5410 0x7f76700f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f7660009fb0 tx=0x7f7660009b30).stop 2024-02-12T16:35:12.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.116+0000 7f767754d700 1 -- 172.21.15.149:0/67855816 shutdown_connections 2024-02-12T16:35:12.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.116+0000 7f767754d700 1 --2- 172.21.15.149:0/67855816 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76700f5410 0x7f76700f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:12.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.116+0000 7f767754d700 1 -- 172.21.15.149:0/67855816 >> 172.21.15.149:0/67855816 conn(0x7f76700f0b70 msgr2=0x7f76700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:12.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.116+0000 7f767754d700 1 -- 172.21.15.149:0/67855816 shutdown_connections 2024-02-12T16:35:12.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.116+0000 7f767754d700 1 -- 172.21.15.149:0/67855816 wait complete. 2024-02-12T16:35:12.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.117+0000 7f767754d700 1 Processor -- start 2024-02-12T16:35:12.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.117+0000 7f767754d700 1 -- start start 2024-02-12T16:35:12.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.117+0000 7f767754d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76700f5410 0x7f76700fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:12.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.117+0000 7f767754d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76700fb960 con 0x7f76700f5410 2024-02-12T16:35:12.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.117+0000 7f76752e9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76700f5410 0x7f76700fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:12.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.117+0000 7f76752e9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76700f5410 0x7f76700fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:39936/0 (socket says 172.21.15.149:39936) 2024-02-12T16:35:12.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.117+0000 7f76752e9700 1 -- 172.21.15.149:0/2052740048 learned_addr learned my addr 172.21.15.149:0/2052740048 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:12.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.117+0000 7f76752e9700 1 -- 172.21.15.149:0/2052740048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7660005040 con 0x7f76700f5410 2024-02-12T16:35:12.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.117+0000 7f76752e9700 1 --2- 172.21.15.149:0/2052740048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76700f5410 0x7f76700fd4e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f76600089f0 tx=0x7f766000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:12.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.117+0000 7f7665ffb700 1 -- 172.21.15.149:0/2052740048 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7660009320 con 0x7f76700f5410 2024-02-12T16:35:12.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.118+0000 7f7665ffb700 1 -- 172.21.15.149:0/2052740048 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7660013380 con 0x7f76700f5410 2024-02-12T16:35:12.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.118+0000 7f767754d700 1 -- 172.21.15.149:0/2052740048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76700fbb00 con 0x7f76700f5410 2024-02-12T16:35:12.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.118+0000 7f7665ffb700 1 -- 172.21.15.149:0/2052740048 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f766001ed80 con 0x7f76700f5410 2024-02-12T16:35:12.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.118+0000 7f767754d700 1 -- 172.21.15.149:0/2052740048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76700fbf90 con 0x7f76700f5410 2024-02-12T16:35:12.121 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.118+0000 7f7665ffb700 1 -- 172.21.15.149:0/2052740048 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f766001e420 con 0x7f76700f5410 2024-02-12T16:35:12.121 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.119+0000 7f767754d700 1 -- 172.21.15.149:0/2052740048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7670059100 con 0x7f76700f5410 2024-02-12T16:35:12.121 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.119+0000 7f7665ffb700 1 --2- 172.21.15.149:0/2052740048 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f765c02ff60 0x7f765c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:12.121 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.119+0000 7f7665ffb700 1 -- 172.21.15.149:0/2052740048 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7660041120 con 0x7f76700f5410 2024-02-12T16:35:12.121 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.119+0000 7f7674ae8700 1 --2- 172.21.15.149:0/2052740048 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f765c02ff60 0x7f765c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:12.121 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.119+0000 7f7674ae8700 1 --2- 172.21.15.149:0/2052740048 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f765c02ff60 0x7f765c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f766c0096a0 tx=0x7f766c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:12.124 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.123+0000 7f7665ffb700 1 -- 172.21.15.149:0/2052740048 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f766001cb10 con 0x7f76700f5410 2024-02-12T16:35:12.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.327+0000 7f767754d700 1 -- 172.21.15.149:0/2052740048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7670045ad0 con 0x7f76700f5410 2024-02-12T16:35:12.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.328+0000 7f7665ffb700 1 -- 172.21.15.149:0/2052740048 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f766002c2d0 con 0x7f76700f5410 2024-02-12T16:35:12.329 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:12.329 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:12.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.329+0000 7f767754d700 1 -- 172.21.15.149:0/2052740048 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f765c02ff60 msgr2=0x7f765c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:12.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.329+0000 7f767754d700 1 --2- 172.21.15.149:0/2052740048 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f765c02ff60 0x7f765c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f766c0096a0 tx=0x7f766c006b40).stop 2024-02-12T16:35:12.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.329+0000 7f767754d700 1 -- 172.21.15.149:0/2052740048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76700f5410 msgr2=0x7f76700fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:12.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.329+0000 7f767754d700 1 --2- 172.21.15.149:0/2052740048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76700f5410 0x7f76700fd4e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f76600089f0 tx=0x7f766000b040).stop 2024-02-12T16:35:12.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.329+0000 7f767754d700 1 -- 172.21.15.149:0/2052740048 shutdown_connections 2024-02-12T16:35:12.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.329+0000 7f767754d700 1 --2- 172.21.15.149:0/2052740048 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f765c02ff60 0x7f765c032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:12.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.329+0000 7f767754d700 1 --2- 172.21.15.149:0/2052740048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76700f5410 0x7f76700fd4e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:12.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.329+0000 7f767754d700 1 -- 172.21.15.149:0/2052740048 >> 172.21.15.149:0/2052740048 conn(0x7f76700f0b70 msgr2=0x7f767005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:12.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.329+0000 7f767754d700 1 -- 172.21.15.149:0/2052740048 shutdown_connections 2024-02-12T16:35:12.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:12.329+0000 7f767754d700 1 -- 172.21.15.149:0/2052740048 wait complete. 2024-02-12T16:35:12.331 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:12.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:10.326185+0000 mgr.smithi104.ohwbhf (mgr.14162) 174 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:12.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:35:10.435080+0000 mgr.smithi104.ohwbhf (mgr.14162) 175 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi104 ... 2024-02-12T16:35:12.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-12T16:35:12.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-12T16:35:12.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:30:16Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:30:54Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:31:32Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-12T16:35:12.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-12T16:35:12.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-12T16:35:12.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:32:47Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:33:24Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:34:32Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-12T16:35:12.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Traceback (most recent call last): 2024-02-12T16:35:12.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-12T16:35:12.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: main() 2024-02-12T16:35:12.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-12T16:35:12.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: r = ctx.func(ctx) 2024-02-12T16:35:12.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-12T16:35:12.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: return func(ctx) 2024-02-12T16:35:12.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-12T16:35:12.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-12T16:35:12.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-12T16:35:12.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-12T16:35:12.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-12T16:35:12.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: raise RuntimeError('uid/gid not found') 2024-02-12T16:35:12.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: RuntimeError: uid/gid not found 2024-02-12T16:35:12.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Traceback (most recent call last): 2024-02-12T16:35:12.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-12T16:35:12.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: yield (conn, connr) 2024-02-12T16:35:12.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-12T16:35:12.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: code, '\n'.join(err))) 2024-02-12T16:35:12.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi104 ... 2024-02-12T16:35:12.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-12T16:35:12.430 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-12T16:35:12.430 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:30:16Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.430 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:30:54Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.430 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:31:32Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.430 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-12T16:35:12.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-12T16:35:12.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-12T16:35:12.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:32:47Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:33:24Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:34:32Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:35:12.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-12T16:35:12.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Traceback (most recent call last): 2024-02-12T16:35:12.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-12T16:35:12.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: main() 2024-02-12T16:35:12.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-12T16:35:12.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: r = ctx.func(ctx) 2024-02-12T16:35:12.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-12T16:35:12.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: return func(ctx) 2024-02-12T16:35:12.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-12T16:35:12.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-12T16:35:12.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-12T16:35:12.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-12T16:35:12.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-12T16:35:12.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: raise RuntimeError('uid/gid not found') 2024-02-12T16:35:12.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: RuntimeError: uid/gid not found 2024-02-12T16:35:12.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:35:10.443448+0000 mgr.smithi104.ohwbhf (mgr.14162) 176 : cephadm [INF] Deploying daemon crash.smithi104 on smithi104 2024-02-12T16:35:13.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:12.328097+0000 mon.smithi104 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.149:0/2052740048' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:13.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:12.806521+0000 mon.smithi104 (mon.0) 215 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:35:13.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:13.126467+0000 mon.smithi104 (mon.0) 216 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:35:13.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:13.128856+0000 mon.smithi104 (mon.0) 217 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:35:13.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:13.129300+0000 mon.smithi104 (mon.0) 218 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-12T16:35:13.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:13.132059+0000 mon.smithi104 (mon.0) 219 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:35:14.004 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:14.004 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:14.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:12.326522+0000 mgr.smithi104.ohwbhf (mgr.14162) 177 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:14.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:13.129735+0000 mgr.smithi104.ohwbhf (mgr.14162) 178 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-12T16:35:14.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:35:13.134919+0000 mgr.smithi104.ohwbhf (mgr.14162) 179 : cephadm [INF] Deploying daemon grafana.smithi104 on smithi104 2024-02-12T16:35:15.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.927+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/885449891 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e380f5410 msgr2=0x7f8e380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:15.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.927+0000 7f8e3f2bc700 1 --2- 172.21.15.149:0/885449891 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e380f5410 0x7f8e380f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f8e34009fb0 tx=0x7f8e34009b30).stop 2024-02-12T16:35:15.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.927+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/885449891 shutdown_connections 2024-02-12T16:35:15.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.927+0000 7f8e3f2bc700 1 --2- 172.21.15.149:0/885449891 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e380f5410 0x7f8e380f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:15.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.927+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/885449891 >> 172.21.15.149:0/885449891 conn(0x7f8e380f0b70 msgr2=0x7f8e380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:15.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.927+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/885449891 shutdown_connections 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.927+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/885449891 wait complete. 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.927+0000 7f8e3f2bc700 1 Processor -- start 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.928+0000 7f8e3f2bc700 1 -- start start 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.928+0000 7f8e3f2bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e380f5410 0x7f8e381056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.928+0000 7f8e3f2bc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e38105bd0 con 0x7f8e380f5410 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.928+0000 7f8e3d058700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e380f5410 0x7f8e381056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.928+0000 7f8e3d058700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e380f5410 0x7f8e381056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:58656/0 (socket says 172.21.15.149:58656) 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.928+0000 7f8e3d058700 1 -- 172.21.15.149:0/3920067351 learned_addr learned my addr 172.21.15.149:0/3920067351 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.928+0000 7f8e3d058700 1 -- 172.21.15.149:0/3920067351 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e34005040 con 0x7f8e380f5410 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.929+0000 7f8e3d058700 1 --2- 172.21.15.149:0/3920067351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e380f5410 0x7f8e381056e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f8e34005010 tx=0x7f8e3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.929+0000 7f8e2dffb700 1 -- 172.21.15.149:0/3920067351 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e34009320 con 0x7f8e380f5410 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.929+0000 7f8e2dffb700 1 -- 172.21.15.149:0/3920067351 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e34013380 con 0x7f8e380f5410 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.929+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/3920067351 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e38105dd0 con 0x7f8e380f5410 2024-02-12T16:35:15.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.929+0000 7f8e2dffb700 1 -- 172.21.15.149:0/3920067351 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e34014e50 con 0x7f8e380f5410 2024-02-12T16:35:15.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.929+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/3920067351 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e38106260 con 0x7f8e380f5410 2024-02-12T16:35:15.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.930+0000 7f8e2dffb700 1 -- 172.21.15.149:0/3920067351 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e340095f0 con 0x7f8e380f5410 2024-02-12T16:35:15.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.930+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/3920067351 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e38045ad0 con 0x7f8e380f5410 2024-02-12T16:35:15.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.930+0000 7f8e2dffb700 1 --2- 172.21.15.149:0/3920067351 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8e2402ff10 0x7f8e240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:15.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.930+0000 7f8e2dffb700 1 -- 172.21.15.149:0/3920067351 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e34040fa0 con 0x7f8e380f5410 2024-02-12T16:35:15.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.930+0000 7f8e3c857700 1 --2- 172.21.15.149:0/3920067351 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8e2402ff10 0x7f8e240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:15.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.931+0000 7f8e3c857700 1 --2- 172.21.15.149:0/3920067351 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8e2402ff10 0x7f8e240323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f8e280096a0 tx=0x7f8e28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:15.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:15.935+0000 7f8e2dffb700 1 -- 172.21.15.149:0/3920067351 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e34014420 con 0x7f8e380f5410 2024-02-12T16:35:16.143 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.142+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/3920067351 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e38107110 con 0x7f8e380f5410 2024-02-12T16:35:16.143 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.142+0000 7f8e2dffb700 1 -- 172.21.15.149:0/3920067351 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8e34010230 con 0x7f8e380f5410 2024-02-12T16:35:16.144 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:16.144 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:16.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.144+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/3920067351 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8e2402ff10 msgr2=0x7f8e240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:16.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.144+0000 7f8e3f2bc700 1 --2- 172.21.15.149:0/3920067351 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8e2402ff10 0x7f8e240323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f8e280096a0 tx=0x7f8e28006b40).stop 2024-02-12T16:35:16.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.144+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/3920067351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e380f5410 msgr2=0x7f8e381056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:16.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.144+0000 7f8e3f2bc700 1 --2- 172.21.15.149:0/3920067351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e380f5410 0x7f8e381056e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f8e34005010 tx=0x7f8e3400b040).stop 2024-02-12T16:35:16.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.144+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/3920067351 shutdown_connections 2024-02-12T16:35:16.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.144+0000 7f8e3f2bc700 1 --2- 172.21.15.149:0/3920067351 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f8e2402ff10 0x7f8e240323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:16.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.144+0000 7f8e3f2bc700 1 --2- 172.21.15.149:0/3920067351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8e380f5410 0x7f8e381056e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:16.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.144+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/3920067351 >> 172.21.15.149:0/3920067351 conn(0x7f8e380f0b70 msgr2=0x7f8e380f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:16.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.144+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/3920067351 shutdown_connections 2024-02-12T16:35:16.146 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:16.144+0000 7f8e3f2bc700 1 -- 172.21.15.149:0/3920067351 wait complete. 2024-02-12T16:35:16.146 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:16.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:14.326897+0000 mgr.smithi104.ohwbhf (mgr.14162) 180 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:16.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:16.142566+0000 mon.smithi104 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.149:0/3920067351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:17.856 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:17.857 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:18.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:16.327199+0000 mgr.smithi104.ohwbhf (mgr.14162) 181 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:19.619 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.617+0000 7fb8d293f700 1 -- 172.21.15.149:0/165811378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 msgr2=0x7fb8cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:19.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.617+0000 7fb8d293f700 1 --2- 172.21.15.149:0/165811378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc0f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fb8c0009fb0 tx=0x7fb8c0009b30).stop 2024-02-12T16:35:19.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.617+0000 7fb8d293f700 1 -- 172.21.15.149:0/165811378 shutdown_connections 2024-02-12T16:35:19.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.617+0000 7fb8d293f700 1 --2- 172.21.15.149:0/165811378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc0f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:19.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.617+0000 7fb8d293f700 1 -- 172.21.15.149:0/165811378 >> 172.21.15.149:0/165811378 conn(0x7fb8cc0f1c70 msgr2=0x7fb8cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:19.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.617+0000 7fb8d293f700 1 -- 172.21.15.149:0/165811378 shutdown_connections 2024-02-12T16:35:19.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.617+0000 7fb8d293f700 1 -- 172.21.15.149:0/165811378 wait complete. 2024-02-12T16:35:19.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.617+0000 7fb8d293f700 1 Processor -- start 2024-02-12T16:35:19.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.618+0000 7fb8d293f700 1 -- start start 2024-02-12T16:35:19.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.618+0000 7fb8d293f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:19.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.618+0000 7fb8d293f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc106cd0 con 0x7fb8cc0f6510 2024-02-12T16:35:19.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.618+0000 7fb8cbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:19.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.618+0000 7fb8cbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:58680/0 (socket says 172.21.15.149:58680) 2024-02-12T16:35:19.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.618+0000 7fb8cbfff700 1 -- 172.21.15.149:0/924441211 learned_addr learned my addr 172.21.15.149:0/924441211 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:19.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.618+0000 7fb8cbfff700 1 -- 172.21.15.149:0/924441211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c0005040 con 0x7fb8cc0f6510 2024-02-12T16:35:19.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.619+0000 7fb8cbfff700 1 --2- 172.21.15.149:0/924441211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fb8c0005010 tx=0x7fb8c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:19.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.619+0000 7fb8c8ff9700 1 -- 172.21.15.149:0/924441211 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c0009320 con 0x7fb8cc0f6510 2024-02-12T16:35:19.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.619+0000 7fb8d293f700 1 -- 172.21.15.149:0/924441211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8cc106ed0 con 0x7fb8cc0f6510 2024-02-12T16:35:19.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.619+0000 7fb8c8ff9700 1 -- 172.21.15.149:0/924441211 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8c0013380 con 0x7fb8cc0f6510 2024-02-12T16:35:19.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.619+0000 7fb8c8ff9700 1 -- 172.21.15.149:0/924441211 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c0014e50 con 0x7fb8cc0f6510 2024-02-12T16:35:19.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.619+0000 7fb8d293f700 1 -- 172.21.15.149:0/924441211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8cc107360 con 0x7fb8cc0f6510 2024-02-12T16:35:19.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.620+0000 7fb8c8ff9700 1 -- 172.21.15.149:0/924441211 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8c00263a0 con 0x7fb8cc0f6510 2024-02-12T16:35:19.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.620+0000 7fb8c8ff9700 1 --2- 172.21.15.149:0/924441211 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 0x7fb8b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:19.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.620+0000 7fb8c8ff9700 1 -- 172.21.15.149:0/924441211 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8c0040e80 con 0x7fb8cc0f6510 2024-02-12T16:35:19.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.620+0000 7fb8d293f700 1 -- 172.21.15.149:0/924441211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8cc045ad0 con 0x7fb8cc0f6510 2024-02-12T16:35:19.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.620+0000 7fb8cb7fe700 1 --2- 172.21.15.149:0/924441211 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 0x7fb8b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:19.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.620+0000 7fb8cb7fe700 1 --2- 172.21.15.149:0/924441211 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 0x7fb8b40323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fb8bc0096a0 tx=0x7fb8bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:19.626 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.625+0000 7fb8c8ff9700 1 -- 172.21.15.149:0/924441211 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8c0012920 con 0x7fb8cc0f6510 2024-02-12T16:35:19.831 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.829+0000 7fb8d293f700 1 -- 172.21.15.149:0/924441211 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8cc108080 con 0x7fb8cc0f6510 2024-02-12T16:35:19.832 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.830+0000 7fb8c8ff9700 1 -- 172.21.15.149:0/924441211 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb8c0012330 con 0x7fb8cc0f6510 2024-02-12T16:35:19.832 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:19.832 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:19.833 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.831+0000 7fb8d293f700 1 -- 172.21.15.149:0/924441211 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 msgr2=0x7fb8b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:19.833 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.831+0000 7fb8d293f700 1 --2- 172.21.15.149:0/924441211 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 0x7fb8b40323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fb8bc0096a0 tx=0x7fb8bc006b40).stop 2024-02-12T16:35:19.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.831+0000 7fb8d293f700 1 -- 172.21.15.149:0/924441211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 msgr2=0x7fb8cc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:19.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.831+0000 7fb8d293f700 1 --2- 172.21.15.149:0/924441211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fb8c0005010 tx=0x7fb8c000b040).stop 2024-02-12T16:35:19.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.831+0000 7fb8d293f700 1 -- 172.21.15.149:0/924441211 shutdown_connections 2024-02-12T16:35:19.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.831+0000 7fb8d293f700 1 --2- 172.21.15.149:0/924441211 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 0x7fb8b40323d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:19.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.831+0000 7fb8d293f700 1 --2- 172.21.15.149:0/924441211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:19.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.831+0000 7fb8d293f700 1 -- 172.21.15.149:0/924441211 >> 172.21.15.149:0/924441211 conn(0x7fb8cc0f1c70 msgr2=0x7fb8cc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:19.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.831+0000 7fb8d293f700 1 -- 172.21.15.149:0/924441211 shutdown_connections 2024-02-12T16:35:19.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:19.831+0000 7fb8d293f700 1 -- 172.21.15.149:0/924441211 wait complete. 2024-02-12T16:35:19.834 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:18.327520+0000 mgr.smithi104.ohwbhf (mgr.14162) 182 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:19.831146+0000 mon.smithi104 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.149:0/924441211' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:21.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:21.474 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:22.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:20.327791+0000 mgr.smithi104.ohwbhf (mgr.14162) 183 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:23.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.193+0000 7f92a13de700 1 -- 172.21.15.149:0/2556369106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f929c0f5410 msgr2=0x7f929c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:23.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.193+0000 7f92a13de700 1 --2- 172.21.15.149:0/2556369106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f929c0f5410 0x7f929c0f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f9290009fb0 tx=0x7f9290009b30).stop 2024-02-12T16:35:23.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.193+0000 7f92a13de700 1 -- 172.21.15.149:0/2556369106 shutdown_connections 2024-02-12T16:35:23.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.193+0000 7f92a13de700 1 --2- 172.21.15.149:0/2556369106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f929c0f5410 0x7f929c0f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:23.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.193+0000 7f92a13de700 1 -- 172.21.15.149:0/2556369106 >> 172.21.15.149:0/2556369106 conn(0x7f929c0f0b70 msgr2=0x7f929c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:23.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.193+0000 7f92a13de700 1 -- 172.21.15.149:0/2556369106 shutdown_connections 2024-02-12T16:35:23.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.193+0000 7f92a13de700 1 -- 172.21.15.149:0/2556369106 wait complete. 2024-02-12T16:35:23.199 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.193+0000 7f92a13de700 1 Processor -- start 2024-02-12T16:35:23.199 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.193+0000 7f92a13de700 1 -- start start 2024-02-12T16:35:23.199 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f92a13de700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f929c0f5410 0x7f929c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:23.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f92a13de700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c0fb960 con 0x7f929c0f5410 2024-02-12T16:35:23.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f929affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f929c0f5410 0x7f929c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:23.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f929affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f929c0f5410 0x7f929c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:58692/0 (socket says 172.21.15.149:58692) 2024-02-12T16:35:23.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f929affd700 1 -- 172.21.15.149:0/3552967067 learned_addr learned my addr 172.21.15.149:0/3552967067 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:23.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f929affd700 1 -- 172.21.15.149:0/3552967067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9290005040 con 0x7f929c0f5410 2024-02-12T16:35:23.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f929affd700 1 --2- 172.21.15.149:0/3552967067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f929c0f5410 0x7f929c0fd4e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f9290005010 tx=0x7f929000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:23.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f92837fe700 1 -- 172.21.15.149:0/3552967067 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9290009320 con 0x7f929c0f5410 2024-02-12T16:35:23.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f92a13de700 1 -- 172.21.15.149:0/3552967067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c0fbb00 con 0x7f929c0f5410 2024-02-12T16:35:23.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f92837fe700 1 -- 172.21.15.149:0/3552967067 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9290013380 con 0x7f929c0f5410 2024-02-12T16:35:23.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f92837fe700 1 -- 172.21.15.149:0/3552967067 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9290014e50 con 0x7f929c0f5410 2024-02-12T16:35:23.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.194+0000 7f92a13de700 1 -- 172.21.15.149:0/3552967067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c0fbf90 con 0x7f929c0f5410 2024-02-12T16:35:23.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.195+0000 7f92837fe700 1 -- 172.21.15.149:0/3552967067 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92900263a0 con 0x7f929c0f5410 2024-02-12T16:35:23.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.195+0000 7f92837fe700 1 --2- 172.21.15.149:0/3552967067 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f928402ff10 0x7f92840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:23.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.195+0000 7f92837fe700 1 -- 172.21.15.149:0/3552967067 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92900095f0 con 0x7f929c0f5410 2024-02-12T16:35:23.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.195+0000 7f92a13de700 1 -- 172.21.15.149:0/3552967067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c059100 con 0x7f929c0f5410 2024-02-12T16:35:23.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.195+0000 7f929a7fc700 1 --2- 172.21.15.149:0/3552967067 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f928402ff10 0x7f92840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:23.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.196+0000 7f929a7fc700 1 --2- 172.21.15.149:0/3552967067 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f928402ff10 0x7f92840323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f928c0096a0 tx=0x7f928c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:23.204 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.202+0000 7f92837fe700 1 -- 172.21.15.149:0/3552967067 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9290012920 con 0x7f929c0f5410 2024-02-12T16:35:23.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.407+0000 7f92a13de700 1 -- 172.21.15.149:0/3552967067 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f929c0fc690 con 0x7f929c0f5410 2024-02-12T16:35:23.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.407+0000 7f92837fe700 1 -- 172.21.15.149:0/3552967067 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f929002b030 con 0x7f929c0f5410 2024-02-12T16:35:23.409 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:23.410 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:23.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.409+0000 7f92a13de700 1 -- 172.21.15.149:0/3552967067 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f928402ff10 msgr2=0x7f92840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:23.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.409+0000 7f92a13de700 1 --2- 172.21.15.149:0/3552967067 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f928402ff10 0x7f92840323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f928c0096a0 tx=0x7f928c006b40).stop 2024-02-12T16:35:23.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.409+0000 7f92a13de700 1 -- 172.21.15.149:0/3552967067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f929c0f5410 msgr2=0x7f929c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:23.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.409+0000 7f92a13de700 1 --2- 172.21.15.149:0/3552967067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f929c0f5410 0x7f929c0fd4e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f9290005010 tx=0x7f929000b040).stop 2024-02-12T16:35:23.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.409+0000 7f92a13de700 1 -- 172.21.15.149:0/3552967067 shutdown_connections 2024-02-12T16:35:23.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.409+0000 7f92a13de700 1 --2- 172.21.15.149:0/3552967067 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f928402ff10 0x7f92840323d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:23.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.409+0000 7f92a13de700 1 --2- 172.21.15.149:0/3552967067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f929c0f5410 0x7f929c0fd4e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:23.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.409+0000 7f92a13de700 1 -- 172.21.15.149:0/3552967067 >> 172.21.15.149:0/3552967067 conn(0x7f929c0f0b70 msgr2=0x7f929c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:23.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.409+0000 7f92a13de700 1 -- 172.21.15.149:0/3552967067 shutdown_connections 2024-02-12T16:35:23.412 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:23.409+0000 7f92a13de700 1 -- 172.21.15.149:0/3552967067 wait complete. 2024-02-12T16:35:23.412 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:22.327988+0000 mgr.smithi104.ohwbhf (mgr.14162) 184 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:23.408618+0000 mon.smithi104 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.149:0/3552967067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:25.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:25.212 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:24.408654+0000 mon.smithi104 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:35:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:24.414291+0000 mon.smithi104 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:35:26.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:24.328233+0000 mgr.smithi104.ohwbhf (mgr.14162) 185 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:27.093 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.091+0000 7f015f2da700 1 -- 172.21.15.149:0/3824338524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01580f6510 msgr2=0x7f01580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:27.093 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.091+0000 7f015f2da700 1 --2- 172.21.15.149:0/3824338524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01580f6510 0x7f01580f6930 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f0154009fb0 tx=0x7f0154009b30).stop 2024-02-12T16:35:27.093 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.091+0000 7f015f2da700 1 -- 172.21.15.149:0/3824338524 shutdown_connections 2024-02-12T16:35:27.093 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.091+0000 7f015f2da700 1 --2- 172.21.15.149:0/3824338524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01580f6510 0x7f01580f6930 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:27.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.091+0000 7f015f2da700 1 -- 172.21.15.149:0/3824338524 >> 172.21.15.149:0/3824338524 conn(0x7f01580f1c70 msgr2=0x7f01580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:27.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.091+0000 7f015f2da700 1 -- 172.21.15.149:0/3824338524 shutdown_connections 2024-02-12T16:35:27.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.091+0000 7f015f2da700 1 -- 172.21.15.149:0/3824338524 wait complete. 2024-02-12T16:35:27.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.091+0000 7f015f2da700 1 Processor -- start 2024-02-12T16:35:27.094 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.091+0000 7f015f2da700 1 -- start start 2024-02-12T16:35:27.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.092+0000 7f015f2da700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01580f6510 0x7f01581067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:27.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.092+0000 7f015f2da700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0158106cd0 con 0x7f01580f6510 2024-02-12T16:35:27.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.092+0000 7f015d076700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01580f6510 0x7f01581067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:27.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.092+0000 7f015d076700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01580f6510 0x7f01581067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:38176/0 (socket says 172.21.15.149:38176) 2024-02-12T16:35:27.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.092+0000 7f015d076700 1 -- 172.21.15.149:0/1213446879 learned_addr learned my addr 172.21.15.149:0/1213446879 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:27.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.092+0000 7f015d076700 1 -- 172.21.15.149:0/1213446879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154005040 con 0x7f01580f6510 2024-02-12T16:35:27.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.092+0000 7f015d076700 1 --2- 172.21.15.149:0/1213446879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01580f6510 0x7f01581067e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0154005010 tx=0x7f015400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:27.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.092+0000 7f014dffb700 1 -- 172.21.15.149:0/1213446879 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0154009320 con 0x7f01580f6510 2024-02-12T16:35:27.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.092+0000 7f014dffb700 1 -- 172.21.15.149:0/1213446879 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0154013380 con 0x7f01580f6510 2024-02-12T16:35:27.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.092+0000 7f015f2da700 1 -- 172.21.15.149:0/1213446879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0158106ed0 con 0x7f01580f6510 2024-02-12T16:35:27.096 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.092+0000 7f014dffb700 1 -- 172.21.15.149:0/1213446879 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0154014e50 con 0x7f01580f6510 2024-02-12T16:35:27.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.093+0000 7f015f2da700 1 -- 172.21.15.149:0/1213446879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0158107360 con 0x7f01580f6510 2024-02-12T16:35:27.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.093+0000 7f014dffb700 1 -- 172.21.15.149:0/1213446879 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0154014420 con 0x7f01580f6510 2024-02-12T16:35:27.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.093+0000 7f015f2da700 1 -- 172.21.15.149:0/1213446879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0158045ad0 con 0x7f01580f6510 2024-02-12T16:35:27.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.093+0000 7f014dffb700 1 --2- 172.21.15.149:0/1213446879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f014402ff10 0x7f01440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:27.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.093+0000 7f014dffb700 1 -- 172.21.15.149:0/1213446879 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0154040f70 con 0x7f01580f6510 2024-02-12T16:35:27.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.094+0000 7f015c875700 1 --2- 172.21.15.149:0/1213446879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f014402ff10 0x7f01440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:27.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.094+0000 7f015c875700 1 --2- 172.21.15.149:0/1213446879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f014402ff10 0x7f01440323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f01480096a0 tx=0x7f0148006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:27.099 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.098+0000 7f014dffb700 1 -- 172.21.15.149:0/1213446879 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01540095f0 con 0x7f01580f6510 2024-02-12T16:35:27.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.301+0000 7f015f2da700 1 -- 172.21.15.149:0/1213446879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0158108210 con 0x7f01580f6510 2024-02-12T16:35:27.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.302+0000 7f014dffb700 1 -- 172.21.15.149:0/1213446879 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f015401db70 con 0x7f01580f6510 2024-02-12T16:35:27.304 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:27.304 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:27.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.304+0000 7f015f2da700 1 -- 172.21.15.149:0/1213446879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f014402ff10 msgr2=0x7f01440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:27.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.304+0000 7f015f2da700 1 --2- 172.21.15.149:0/1213446879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f014402ff10 0x7f01440323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f01480096a0 tx=0x7f0148006b40).stop 2024-02-12T16:35:27.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.304+0000 7f015f2da700 1 -- 172.21.15.149:0/1213446879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01580f6510 msgr2=0x7f01581067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:27.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.304+0000 7f015f2da700 1 --2- 172.21.15.149:0/1213446879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01580f6510 0x7f01581067e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0154005010 tx=0x7f015400b040).stop 2024-02-12T16:35:27.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.304+0000 7f015f2da700 1 -- 172.21.15.149:0/1213446879 shutdown_connections 2024-02-12T16:35:27.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.304+0000 7f015f2da700 1 --2- 172.21.15.149:0/1213446879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f014402ff10 0x7f01440323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:27.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.304+0000 7f015f2da700 1 --2- 172.21.15.149:0/1213446879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01580f6510 0x7f01581067e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:27.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.304+0000 7f015f2da700 1 -- 172.21.15.149:0/1213446879 >> 172.21.15.149:0/1213446879 conn(0x7f01580f1c70 msgr2=0x7f01580f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:27.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.304+0000 7f015f2da700 1 -- 172.21.15.149:0/1213446879 shutdown_connections 2024-02-12T16:35:27.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:27.304+0000 7f015f2da700 1 -- 172.21.15.149:0/1213446879 wait complete. 2024-02-12T16:35:27.307 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:28.452 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:26.328473+0000 mgr.smithi104.ohwbhf (mgr.14162) 186 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:28.452 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:27.303415+0000 mon.smithi104 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.149:0/1213446879' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:28.971 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:28.971 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:30.448 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:28.328839+0000 mgr.smithi104.ohwbhf (mgr.14162) 187 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:30.803 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.797+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1871344921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a780f6510 msgr2=0x7f3a780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:30.803 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.797+0000 7f3a7de7b700 1 --2- 172.21.15.149:0/1871344921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a780f6510 0x7f3a780f6930 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f3a6c009fb0 tx=0x7f3a6c009b30).stop 2024-02-12T16:35:30.803 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.797+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1871344921 shutdown_connections 2024-02-12T16:35:30.803 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.797+0000 7f3a7de7b700 1 --2- 172.21.15.149:0/1871344921 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a780f6510 0x7f3a780f6930 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:30.804 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.797+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1871344921 >> 172.21.15.149:0/1871344921 conn(0x7f3a780f1c70 msgr2=0x7f3a780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:30.804 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.797+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1871344921 shutdown_connections 2024-02-12T16:35:30.804 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.797+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1871344921 wait complete. 2024-02-12T16:35:30.804 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.798+0000 7f3a7de7b700 1 Processor -- start 2024-02-12T16:35:30.804 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.798+0000 7f3a7de7b700 1 -- start start 2024-02-12T16:35:30.804 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.798+0000 7f3a7de7b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a780f6510 0x7f3a7806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:30.804 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.798+0000 7f3a7de7b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a78069e20 con 0x7f3a780f6510 2024-02-12T16:35:30.804 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.798+0000 7f3a777fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a780f6510 0x7f3a7806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:30.804 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.798+0000 7f3a777fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a780f6510 0x7f3a7806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:38190/0 (socket says 172.21.15.149:38190) 2024-02-12T16:35:30.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.798+0000 7f3a777fe700 1 -- 172.21.15.149:0/1366152151 learned_addr learned my addr 172.21.15.149:0/1366152151 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:30.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.798+0000 7f3a777fe700 1 -- 172.21.15.149:0/1366152151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c005040 con 0x7f3a780f6510 2024-02-12T16:35:30.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.799+0000 7f3a777fe700 1 --2- 172.21.15.149:0/1366152151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a780f6510 0x7f3a7806b9a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f3a6c005010 tx=0x7f3a6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:30.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.799+0000 7f3a5ffff700 1 -- 172.21.15.149:0/1366152151 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a6c009320 con 0x7f3a780f6510 2024-02-12T16:35:30.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.799+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1366152151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a78069fc0 con 0x7f3a780f6510 2024-02-12T16:35:30.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.799+0000 7f3a5ffff700 1 -- 172.21.15.149:0/1366152151 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a6c013380 con 0x7f3a780f6510 2024-02-12T16:35:30.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.799+0000 7f3a5ffff700 1 -- 172.21.15.149:0/1366152151 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a6c014e50 con 0x7f3a780f6510 2024-02-12T16:35:30.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.799+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1366152151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7806a450 con 0x7f3a780f6510 2024-02-12T16:35:30.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.800+0000 7f3a5ffff700 1 -- 172.21.15.149:0/1366152151 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a6c014420 con 0x7f3a780f6510 2024-02-12T16:35:30.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.800+0000 7f3a5ffff700 1 --2- 172.21.15.149:0/1366152151 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3a6002ff10 0x7f3a600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:30.805 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.800+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1366152151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a78059100 con 0x7f3a780f6510 2024-02-12T16:35:30.806 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.800+0000 7f3a5ffff700 1 -- 172.21.15.149:0/1366152151 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a6c040f40 con 0x7f3a780f6510 2024-02-12T16:35:30.806 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.800+0000 7f3a76ffd700 1 --2- 172.21.15.149:0/1366152151 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3a6002ff10 0x7f3a600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:30.806 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.800+0000 7f3a76ffd700 1 --2- 172.21.15.149:0/1366152151 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3a6002ff10 0x7f3a600323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f3a680096a0 tx=0x7f3a68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:30.806 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:30.804+0000 7f3a5ffff700 1 -- 172.21.15.149:0/1366152151 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a6c0095f0 con 0x7f3a780f6510 2024-02-12T16:35:31.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.008+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1366152151 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a780fc0a0 con 0x7f3a780f6510 2024-02-12T16:35:31.011 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.009+0000 7f3a5ffff700 1 -- 172.21.15.149:0/1366152151 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3a6c02b020 con 0x7f3a780f6510 2024-02-12T16:35:31.011 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:31.011 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:31.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.011+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1366152151 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3a6002ff10 msgr2=0x7f3a600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:31.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.011+0000 7f3a7de7b700 1 --2- 172.21.15.149:0/1366152151 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3a6002ff10 0x7f3a600323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f3a680096a0 tx=0x7f3a68006b40).stop 2024-02-12T16:35:31.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.011+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1366152151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a780f6510 msgr2=0x7f3a7806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:31.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.011+0000 7f3a7de7b700 1 --2- 172.21.15.149:0/1366152151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a780f6510 0x7f3a7806b9a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f3a6c005010 tx=0x7f3a6c00b040).stop 2024-02-12T16:35:31.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.011+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1366152151 shutdown_connections 2024-02-12T16:35:31.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.011+0000 7f3a7de7b700 1 --2- 172.21.15.149:0/1366152151 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3a6002ff10 0x7f3a600323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:31.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.011+0000 7f3a7de7b700 1 --2- 172.21.15.149:0/1366152151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a780f6510 0x7f3a7806b9a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:31.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.011+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1366152151 >> 172.21.15.149:0/1366152151 conn(0x7f3a780f1c70 msgr2=0x7f3a780f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:31.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.011+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1366152151 shutdown_connections 2024-02-12T16:35:31.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:31.011+0000 7f3a7de7b700 1 -- 172.21.15.149:0/1366152151 wait complete. 2024-02-12T16:35:31.014 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:31.501 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:31 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:31.010455+0000 mon.smithi104 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.149:0/1366152151' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:32.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:30.329277+0000 mgr.smithi104.ohwbhf (mgr.14162) 188 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:32.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:31.677674+0000 mon.smithi104 (mon.0) 227 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:35:32.660 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:32.661 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:33.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:35:31.678952+0000 mgr.smithi104.ohwbhf (mgr.14162) 189 : cephadm [INF] Deploying daemon node-exporter.smithi104 on smithi104 2024-02-12T16:35:34.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.383+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/3809936602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ce40f5410 msgr2=0x7f9ce40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:34.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.383+0000 7f9ceaf9c700 1 --2- 172.21.15.149:0/3809936602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ce40f5410 0x7f9ce40f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f9cd8009fb0 tx=0x7f9cd8009b30).stop 2024-02-12T16:35:34.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.383+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/3809936602 shutdown_connections 2024-02-12T16:35:34.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.383+0000 7f9ceaf9c700 1 --2- 172.21.15.149:0/3809936602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ce40f5410 0x7f9ce40f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:34.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.383+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/3809936602 >> 172.21.15.149:0/3809936602 conn(0x7f9ce40f0b70 msgr2=0x7f9ce40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:34.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.383+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/3809936602 shutdown_connections 2024-02-12T16:35:34.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.383+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/3809936602 wait complete. 2024-02-12T16:35:34.402 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.384+0000 7f9ceaf9c700 1 Processor -- start 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.384+0000 7f9ceaf9c700 1 -- start start 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.384+0000 7f9ceaf9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ce40f5410 0x7f9ce41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.384+0000 7f9ceaf9c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4105bd0 con 0x7f9ce40f5410 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.384+0000 7f9ce8d38700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ce40f5410 0x7f9ce41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.384+0000 7f9ce8d38700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ce40f5410 0x7f9ce41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:51630/0 (socket says 172.21.15.149:51630) 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.384+0000 7f9ce8d38700 1 -- 172.21.15.149:0/827214002 learned_addr learned my addr 172.21.15.149:0/827214002 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.385+0000 7f9ce8d38700 1 -- 172.21.15.149:0/827214002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd8005040 con 0x7f9ce40f5410 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.385+0000 7f9ce8d38700 1 --2- 172.21.15.149:0/827214002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ce40f5410 0x7f9ce41056e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f9cd8005010 tx=0x7f9cd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.385+0000 7f9ce17fa700 1 -- 172.21.15.149:0/827214002 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd8009320 con 0x7f9ce40f5410 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.385+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/827214002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce4105dd0 con 0x7f9ce40f5410 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.385+0000 7f9ce17fa700 1 -- 172.21.15.149:0/827214002 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cd8013380 con 0x7f9ce40f5410 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.385+0000 7f9ce17fa700 1 -- 172.21.15.149:0/827214002 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd8014e50 con 0x7f9ce40f5410 2024-02-12T16:35:34.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.385+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/827214002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce4106260 con 0x7f9ce40f5410 2024-02-12T16:35:34.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.386+0000 7f9ce17fa700 1 -- 172.21.15.149:0/827214002 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cd80263a0 con 0x7f9ce40f5410 2024-02-12T16:35:34.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.386+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/827214002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce4045ad0 con 0x7f9ce40f5410 2024-02-12T16:35:34.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.386+0000 7f9ce17fa700 1 --2- 172.21.15.149:0/827214002 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9cd402ff10 0x7f9cd40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:34.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.386+0000 7f9ce17fa700 1 -- 172.21.15.149:0/827214002 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cd8040f10 con 0x7f9ce40f5410 2024-02-12T16:35:34.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.386+0000 7f9ce3fff700 1 --2- 172.21.15.149:0/827214002 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9cd402ff10 0x7f9cd40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:34.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.387+0000 7f9ce3fff700 1 --2- 172.21.15.149:0/827214002 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9cd402ff10 0x7f9cd40323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f9cd00096a0 tx=0x7f9cd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:34.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.391+0000 7f9ce17fa700 1 -- 172.21.15.149:0/827214002 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cd80126f0 con 0x7f9ce40f5410 2024-02-12T16:35:34.600 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.598+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/827214002 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ce4107110 con 0x7f9ce40f5410 2024-02-12T16:35:34.600 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.599+0000 7f9ce17fa700 1 -- 172.21.15.149:0/827214002 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9cd8012330 con 0x7f9ce40f5410 2024-02-12T16:35:34.601 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:34.601 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:34.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.600+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/827214002 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9cd402ff10 msgr2=0x7f9cd40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:34.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.600+0000 7f9ceaf9c700 1 --2- 172.21.15.149:0/827214002 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9cd402ff10 0x7f9cd40323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f9cd00096a0 tx=0x7f9cd0006b40).stop 2024-02-12T16:35:34.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.600+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/827214002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ce40f5410 msgr2=0x7f9ce41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:34.603 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.600+0000 7f9ceaf9c700 1 --2- 172.21.15.149:0/827214002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ce40f5410 0x7f9ce41056e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f9cd8005010 tx=0x7f9cd800b040).stop 2024-02-12T16:35:34.603 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.600+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/827214002 shutdown_connections 2024-02-12T16:35:34.603 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.600+0000 7f9ceaf9c700 1 --2- 172.21.15.149:0/827214002 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9cd402ff10 0x7f9cd40323d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:34.603 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.600+0000 7f9ceaf9c700 1 --2- 172.21.15.149:0/827214002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9ce40f5410 0x7f9ce41056e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:34.603 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.600+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/827214002 >> 172.21.15.149:0/827214002 conn(0x7f9ce40f0b70 msgr2=0x7f9ce40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:34.603 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.600+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/827214002 shutdown_connections 2024-02-12T16:35:34.603 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:34.600+0000 7f9ceaf9c700 1 -- 172.21.15.149:0/827214002 wait complete. 2024-02-12T16:35:34.603 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:34.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:32.329510+0000 mgr.smithi104.ohwbhf (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:35.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:35 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:34.599896+0000 mon.smithi104 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.149:0/827214002' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:36.218 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:36.218 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:36.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:34.329810+0000 mgr.smithi104.ohwbhf (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:38.121 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.117+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/848002291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d280f78e0 msgr2=0x7f2d280f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:38.121 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.117+0000 7f2d2e5a1700 1 --2- 172.21.15.149:0/848002291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d280f78e0 0x7f2d280f7d00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f2d18009fb0 tx=0x7f2d18009b30).stop 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.117+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/848002291 shutdown_connections 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.117+0000 7f2d2e5a1700 1 --2- 172.21.15.149:0/848002291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d280f78e0 0x7f2d280f7d00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.117+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/848002291 >> 172.21.15.149:0/848002291 conn(0x7f2d280f0120 msgr2=0x7f2d280f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.117+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/848002291 shutdown_connections 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.117+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/848002291 wait complete. 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.118+0000 7f2d2e5a1700 1 Processor -- start 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.118+0000 7f2d2e5a1700 1 -- start start 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.118+0000 7f2d2e5a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d280f78e0 0x7f2d281065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.118+0000 7f2d2e5a1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d28106ad0 con 0x7f2d280f78e0 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.118+0000 7f2d2d59f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d280f78e0 0x7f2d281065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.118+0000 7f2d2d59f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d280f78e0 0x7f2d281065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:51648/0 (socket says 172.21.15.149:51648) 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.118+0000 7f2d2d59f700 1 -- 172.21.15.149:0/1101086649 learned_addr learned my addr 172.21.15.149:0/1101086649 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.119+0000 7f2d2d59f700 1 -- 172.21.15.149:0/1101086649 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18005040 con 0x7f2d280f78e0 2024-02-12T16:35:38.123 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.119+0000 7f2d2d59f700 1 --2- 172.21.15.149:0/1101086649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d280f78e0 0x7f2d281065e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f2d180089f0 tx=0x7f2d1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:38.123 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.119+0000 7f2d1dffb700 1 -- 172.21.15.149:0/1101086649 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d18009320 con 0x7f2d280f78e0 2024-02-12T16:35:38.123 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.119+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/1101086649 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d28106cd0 con 0x7f2d280f78e0 2024-02-12T16:35:38.123 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.119+0000 7f2d1dffb700 1 -- 172.21.15.149:0/1101086649 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d18013380 con 0x7f2d280f78e0 2024-02-12T16:35:38.123 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.119+0000 7f2d1dffb700 1 -- 172.21.15.149:0/1101086649 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d18014e20 con 0x7f2d280f78e0 2024-02-12T16:35:38.123 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.119+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/1101086649 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d28107160 con 0x7f2d280f78e0 2024-02-12T16:35:38.124 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.120+0000 7f2d1dffb700 1 -- 172.21.15.149:0/1101086649 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d180095f0 con 0x7f2d280f78e0 2024-02-12T16:35:38.124 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.120+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/1101086649 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d28045ad0 con 0x7f2d280f78e0 2024-02-12T16:35:38.124 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.120+0000 7f2d1dffb700 1 --2- 172.21.15.149:0/1101086649 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2d1402ff10 0x7f2d140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:38.124 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.120+0000 7f2d1dffb700 1 -- 172.21.15.149:0/1101086649 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d18040f50 con 0x7f2d280f78e0 2024-02-12T16:35:38.124 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.120+0000 7f2d2cd9e700 1 --2- 172.21.15.149:0/1101086649 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2d1402ff10 0x7f2d140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:38.124 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.121+0000 7f2d2cd9e700 1 --2- 172.21.15.149:0/1101086649 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2d1402ff10 0x7f2d140323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f2d240096a0 tx=0x7f2d24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:38.126 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.125+0000 7f2d1dffb700 1 -- 172.21.15.149:0/1101086649 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d18014420 con 0x7f2d280f78e0 2024-02-12T16:35:38.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.329+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/1101086649 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d28108010 con 0x7f2d280f78e0 2024-02-12T16:35:38.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.330+0000 7f2d1dffb700 1 -- 172.21.15.149:0/1101086649 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2d1801d980 con 0x7f2d280f78e0 2024-02-12T16:35:38.332 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:38.332 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:38.333 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.331+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/1101086649 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2d1402ff10 msgr2=0x7f2d140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:38.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.331+0000 7f2d2e5a1700 1 --2- 172.21.15.149:0/1101086649 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2d1402ff10 0x7f2d140323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f2d240096a0 tx=0x7f2d24006b40).stop 2024-02-12T16:35:38.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.331+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/1101086649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d280f78e0 msgr2=0x7f2d281065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:38.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.331+0000 7f2d2e5a1700 1 --2- 172.21.15.149:0/1101086649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d280f78e0 0x7f2d281065e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f2d180089f0 tx=0x7f2d1800b040).stop 2024-02-12T16:35:38.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.331+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/1101086649 shutdown_connections 2024-02-12T16:35:38.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.331+0000 7f2d2e5a1700 1 --2- 172.21.15.149:0/1101086649 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2d1402ff10 0x7f2d140323d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:38.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.331+0000 7f2d2e5a1700 1 --2- 172.21.15.149:0/1101086649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d280f78e0 0x7f2d281065e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:38.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.332+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/1101086649 >> 172.21.15.149:0/1101086649 conn(0x7f2d280f0120 msgr2=0x7f2d280f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:38.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.332+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/1101086649 shutdown_connections 2024-02-12T16:35:38.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:38.332+0000 7f2d2e5a1700 1 -- 172.21.15.149:0/1101086649 wait complete. 2024-02-12T16:35:38.335 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:38.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:36.330087+0000 mgr.smithi104.ohwbhf (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:39.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:39 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:38.331306+0000 mon.smithi104 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.149:0/1101086649' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:40.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:40.090 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:40.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:38.330447+0000 mgr.smithi104.ohwbhf (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:41.984 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.981+0000 7f11022f1700 1 -- 172.21.15.149:0/3062704818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10fc060280 msgr2=0x7f10fc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:41.984 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.981+0000 7f11022f1700 1 --2- 172.21.15.149:0/3062704818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10fc060280 0x7f10fc0f90f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f10ec009fb0 tx=0x7f10ec009b30).stop 2024-02-12T16:35:41.984 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.981+0000 7f11022f1700 1 -- 172.21.15.149:0/3062704818 shutdown_connections 2024-02-12T16:35:41.984 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.981+0000 7f11022f1700 1 --2- 172.21.15.149:0/3062704818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10fc060280 0x7f10fc0f90f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:41.984 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.981+0000 7f11022f1700 1 -- 172.21.15.149:0/3062704818 >> 172.21.15.149:0/3062704818 conn(0x7f10fc0f0b70 msgr2=0x7f10fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:41.984 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.981+0000 7f11022f1700 1 -- 172.21.15.149:0/3062704818 shutdown_connections 2024-02-12T16:35:41.984 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.981+0000 7f11022f1700 1 -- 172.21.15.149:0/3062704818 wait complete. 2024-02-12T16:35:41.984 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.982+0000 7f11022f1700 1 Processor -- start 2024-02-12T16:35:41.984 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.982+0000 7f11022f1700 1 -- start start 2024-02-12T16:35:41.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.982+0000 7f11022f1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10fc060280 0x7f10fc0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:41.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.982+0000 7f11022f1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc0f4ff0 con 0x7f10fc060280 2024-02-12T16:35:41.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.982+0000 7f10fbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10fc060280 0x7f10fc0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:41.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.982+0000 7f10fbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10fc060280 0x7f10fc0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:51664/0 (socket says 172.21.15.149:51664) 2024-02-12T16:35:41.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.982+0000 7f10fbfff700 1 -- 172.21.15.149:0/2656689355 learned_addr learned my addr 172.21.15.149:0/2656689355 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:41.986 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.983+0000 7f10fbfff700 1 -- 172.21.15.149:0/2656689355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec005040 con 0x7f10fc060280 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.983+0000 7f10fbfff700 1 --2- 172.21.15.149:0/2656689355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10fc060280 0x7f10fc0f6b70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f10ec0089f0 tx=0x7f10ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.983+0000 7f10f8ff9700 1 -- 172.21.15.149:0/2656689355 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ec009320 con 0x7f10fc060280 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.983+0000 7f11022f1700 1 -- 172.21.15.149:0/2656689355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc0f5190 con 0x7f10fc060280 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.983+0000 7f10f8ff9700 1 -- 172.21.15.149:0/2656689355 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10ec013380 con 0x7f10fc060280 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.983+0000 7f10f8ff9700 1 -- 172.21.15.149:0/2656689355 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ec014e20 con 0x7f10fc060280 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.983+0000 7f11022f1700 1 -- 172.21.15.149:0/2656689355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc0f5620 con 0x7f10fc060280 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.984+0000 7f10f8ff9700 1 -- 172.21.15.149:0/2656689355 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10ec014420 con 0x7f10fc060280 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.984+0000 7f11022f1700 1 -- 172.21.15.149:0/2656689355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc059100 con 0x7f10fc060280 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.984+0000 7f10f8ff9700 1 --2- 172.21.15.149:0/2656689355 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10e402ff10 0x7f10e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.984+0000 7f10f8ff9700 1 -- 172.21.15.149:0/2656689355 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10ec040fb0 con 0x7f10fc060280 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.984+0000 7f10fb7fe700 1 --2- 172.21.15.149:0/2656689355 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10e402ff10 0x7f10e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:41.987 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.984+0000 7f10fb7fe700 1 --2- 172.21.15.149:0/2656689355 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10e402ff10 0x7f10e40323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f10f00096a0 tx=0x7f10f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:41.990 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:41.988+0000 7f10f8ff9700 1 -- 172.21.15.149:0/2656689355 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10ec06d2b0 con 0x7f10fc060280 2024-02-12T16:35:42.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.192+0000 7f11022f1700 1 -- 172.21.15.149:0/2656689355 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10fc045ad0 con 0x7f10fc060280 2024-02-12T16:35:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.193+0000 7f10f8ff9700 1 -- 172.21.15.149:0/2656689355 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f10ec014790 con 0x7f10fc060280 2024-02-12T16:35:42.196 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:42.196 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:42.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.195+0000 7f11022f1700 1 -- 172.21.15.149:0/2656689355 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10e402ff10 msgr2=0x7f10e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:42.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.195+0000 7f11022f1700 1 --2- 172.21.15.149:0/2656689355 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10e402ff10 0x7f10e40323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f10f00096a0 tx=0x7f10f0006b40).stop 2024-02-12T16:35:42.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.195+0000 7f11022f1700 1 -- 172.21.15.149:0/2656689355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10fc060280 msgr2=0x7f10fc0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:42.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.195+0000 7f11022f1700 1 --2- 172.21.15.149:0/2656689355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10fc060280 0x7f10fc0f6b70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f10ec0089f0 tx=0x7f10ec00b040).stop 2024-02-12T16:35:42.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.195+0000 7f11022f1700 1 -- 172.21.15.149:0/2656689355 shutdown_connections 2024-02-12T16:35:42.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.195+0000 7f11022f1700 1 --2- 172.21.15.149:0/2656689355 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10e402ff10 0x7f10e40323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:42.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.195+0000 7f11022f1700 1 --2- 172.21.15.149:0/2656689355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10fc060280 0x7f10fc0f6b70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:42.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.195+0000 7f11022f1700 1 -- 172.21.15.149:0/2656689355 >> 172.21.15.149:0/2656689355 conn(0x7f10fc0f0b70 msgr2=0x7f10fc0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:42.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.196+0000 7f11022f1700 1 -- 172.21.15.149:0/2656689355 shutdown_connections 2024-02-12T16:35:42.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:42.196+0000 7f11022f1700 1 -- 172.21.15.149:0/2656689355 wait complete. 2024-02-12T16:35:42.198 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:42.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:40.330828+0000 mgr.smithi104.ohwbhf (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:42.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:42.194617+0000 mon.smithi104 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.149:0/2656689355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:43.908 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:43.908 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:44.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:42.331219+0000 mgr.smithi104.ohwbhf (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:45.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.692+0000 7f064e63a700 1 -- 172.21.15.149:0/1521189381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06480f62a0 msgr2=0x7f06480f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:45.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.692+0000 7f064e63a700 1 --2- 172.21.15.149:0/1521189381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06480f62a0 0x7f06480f66c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f0638009fb0 tx=0x7f0638009b30).stop 2024-02-12T16:35:45.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.692+0000 7f064e63a700 1 -- 172.21.15.149:0/1521189381 shutdown_connections 2024-02-12T16:35:45.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.692+0000 7f064e63a700 1 --2- 172.21.15.149:0/1521189381 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06480f62a0 0x7f06480f66c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:45.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.692+0000 7f064e63a700 1 -- 172.21.15.149:0/1521189381 >> 172.21.15.149:0/1521189381 conn(0x7f06480f1aa0 msgr2=0x7f06480f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:45.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.693+0000 7f064e63a700 1 -- 172.21.15.149:0/1521189381 shutdown_connections 2024-02-12T16:35:45.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.693+0000 7f064e63a700 1 -- 172.21.15.149:0/1521189381 wait complete. 2024-02-12T16:35:45.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.693+0000 7f064e63a700 1 Processor -- start 2024-02-12T16:35:45.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.693+0000 7f064e63a700 1 -- start start 2024-02-12T16:35:45.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.694+0000 7f064e63a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06480f62a0 0x7f06481065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:45.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.694+0000 7f064e63a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0648106ac0 con 0x7f06480f62a0 2024-02-12T16:35:45.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.694+0000 7f064d638700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06480f62a0 0x7f06481065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:45.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.694+0000 7f064d638700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06480f62a0 0x7f06481065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:47234/0 (socket says 172.21.15.149:47234) 2024-02-12T16:35:45.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.694+0000 7f064d638700 1 -- 172.21.15.149:0/2182033073 learned_addr learned my addr 172.21.15.149:0/2182033073 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:45.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.694+0000 7f064d638700 1 -- 172.21.15.149:0/2182033073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0638005040 con 0x7f06480f62a0 2024-02-12T16:35:45.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.694+0000 7f064d638700 1 --2- 172.21.15.149:0/2182033073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06480f62a0 0x7f06481065d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f06380089f0 tx=0x7f063800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:45.697 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.694+0000 7f063dffb700 1 -- 172.21.15.149:0/2182033073 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0638009320 con 0x7f06480f62a0 2024-02-12T16:35:45.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.694+0000 7f063dffb700 1 -- 172.21.15.149:0/2182033073 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0638013380 con 0x7f06480f62a0 2024-02-12T16:35:45.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.694+0000 7f063dffb700 1 -- 172.21.15.149:0/2182033073 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0638014e20 con 0x7f06480f62a0 2024-02-12T16:35:45.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.694+0000 7f064e63a700 1 -- 172.21.15.149:0/2182033073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0648106cc0 con 0x7f06480f62a0 2024-02-12T16:35:45.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.695+0000 7f064e63a700 1 -- 172.21.15.149:0/2182033073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0648107150 con 0x7f06480f62a0 2024-02-12T16:35:45.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.695+0000 7f063dffb700 1 -- 172.21.15.149:0/2182033073 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06380095f0 con 0x7f06480f62a0 2024-02-12T16:35:45.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.695+0000 7f064e63a700 1 -- 172.21.15.149:0/2182033073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0648045ad0 con 0x7f06480f62a0 2024-02-12T16:35:45.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.696+0000 7f063dffb700 1 --2- 172.21.15.149:0/2182033073 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063402ff10 0x7f06340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:45.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.696+0000 7f063dffb700 1 -- 172.21.15.149:0/2182033073 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0638040f50 con 0x7f06480f62a0 2024-02-12T16:35:45.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.696+0000 7f064ce37700 1 --2- 172.21.15.149:0/2182033073 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063402ff10 0x7f06340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:45.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.696+0000 7f064ce37700 1 --2- 172.21.15.149:0/2182033073 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063402ff10 0x7f06340323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f06440096a0 tx=0x7f0644006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:45.702 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.700+0000 7f063dffb700 1 -- 172.21.15.149:0/2182033073 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0638014420 con 0x7f06480f62a0 2024-02-12T16:35:45.905 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.903+0000 7f064e63a700 1 -- 172.21.15.149:0/2182033073 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0648108000 con 0x7f06480f62a0 2024-02-12T16:35:45.905 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.903+0000 7f063dffb700 1 -- 172.21.15.149:0/2182033073 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f063801d980 con 0x7f06480f62a0 2024-02-12T16:35:45.905 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:45.905 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:45.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.905+0000 7f064e63a700 1 -- 172.21.15.149:0/2182033073 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063402ff10 msgr2=0x7f06340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:45.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.905+0000 7f064e63a700 1 --2- 172.21.15.149:0/2182033073 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063402ff10 0x7f06340323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f06440096a0 tx=0x7f0644006b40).stop 2024-02-12T16:35:45.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.905+0000 7f064e63a700 1 -- 172.21.15.149:0/2182033073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06480f62a0 msgr2=0x7f06481065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:45.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.905+0000 7f064e63a700 1 --2- 172.21.15.149:0/2182033073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06480f62a0 0x7f06481065d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f06380089f0 tx=0x7f063800b040).stop 2024-02-12T16:35:45.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.905+0000 7f064e63a700 1 -- 172.21.15.149:0/2182033073 shutdown_connections 2024-02-12T16:35:45.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.905+0000 7f064e63a700 1 --2- 172.21.15.149:0/2182033073 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063402ff10 0x7f06340323d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:45.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.905+0000 7f064e63a700 1 --2- 172.21.15.149:0/2182033073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06480f62a0 0x7f06481065d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:45.907 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.905+0000 7f064e63a700 1 -- 172.21.15.149:0/2182033073 >> 172.21.15.149:0/2182033073 conn(0x7f06480f1aa0 msgr2=0x7f06480f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:45.908 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.905+0000 7f064e63a700 1 -- 172.21.15.149:0/2182033073 shutdown_connections 2024-02-12T16:35:45.908 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:45.905+0000 7f064e63a700 1 -- 172.21.15.149:0/2182033073 wait complete. 2024-02-12T16:35:45.908 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:46.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:44.331583+0000 mgr.smithi104.ohwbhf (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:46.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:45.904675+0000 mon.smithi104 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.149:0/2182033073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:47.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:47.539 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:48.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:46.331849+0000 mgr.smithi104.ohwbhf (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:49.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.403+0000 7fc2347f8700 1 -- 172.21.15.149:0/2990544386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c0f5410 msgr2=0x7fc22c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:49.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.403+0000 7fc2347f8700 1 --2- 172.21.15.149:0/2990544386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c0f5410 0x7fc22c0f5830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fc224009fb0 tx=0x7fc224009b30).stop 2024-02-12T16:35:49.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.403+0000 7fc2347f8700 1 -- 172.21.15.149:0/2990544386 shutdown_connections 2024-02-12T16:35:49.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.403+0000 7fc2347f8700 1 --2- 172.21.15.149:0/2990544386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c0f5410 0x7fc22c0f5830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:49.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.403+0000 7fc2347f8700 1 -- 172.21.15.149:0/2990544386 >> 172.21.15.149:0/2990544386 conn(0x7fc22c0f0b70 msgr2=0x7fc22c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:49.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.403+0000 7fc2347f8700 1 -- 172.21.15.149:0/2990544386 shutdown_connections 2024-02-12T16:35:49.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.403+0000 7fc2347f8700 1 -- 172.21.15.149:0/2990544386 wait complete. 2024-02-12T16:35:49.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.404+0000 7fc2347f8700 1 Processor -- start 2024-02-12T16:35:49.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.404+0000 7fc2347f8700 1 -- start start 2024-02-12T16:35:49.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.404+0000 7fc2347f8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c0f5410 0x7fc22c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:49.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.404+0000 7fc2347f8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c105bd0 con 0x7fc22c0f5410 2024-02-12T16:35:49.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.404+0000 7fc232594700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c0f5410 0x7fc22c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:49.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.404+0000 7fc232594700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c0f5410 0x7fc22c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:47258/0 (socket says 172.21.15.149:47258) 2024-02-12T16:35:49.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.404+0000 7fc232594700 1 -- 172.21.15.149:0/70524577 learned_addr learned my addr 172.21.15.149:0/70524577 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:49.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.404+0000 7fc232594700 1 -- 172.21.15.149:0/70524577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc224005040 con 0x7fc22c0f5410 2024-02-12T16:35:49.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.405+0000 7fc232594700 1 --2- 172.21.15.149:0/70524577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c0f5410 0x7fc22c1056e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fc2240089f0 tx=0x7fc22400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:49.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.405+0000 7fc22affd700 1 -- 172.21.15.149:0/70524577 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc224009320 con 0x7fc22c0f5410 2024-02-12T16:35:49.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.405+0000 7fc2347f8700 1 -- 172.21.15.149:0/70524577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22c105dd0 con 0x7fc22c0f5410 2024-02-12T16:35:49.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.405+0000 7fc22affd700 1 -- 172.21.15.149:0/70524577 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc224013380 con 0x7fc22c0f5410 2024-02-12T16:35:49.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.405+0000 7fc22affd700 1 -- 172.21.15.149:0/70524577 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc22401ed80 con 0x7fc22c0f5410 2024-02-12T16:35:49.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.405+0000 7fc2347f8700 1 -- 172.21.15.149:0/70524577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22c106260 con 0x7fc22c0f5410 2024-02-12T16:35:49.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.405+0000 7fc22affd700 1 -- 172.21.15.149:0/70524577 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc22401e420 con 0x7fc22c0f5410 2024-02-12T16:35:49.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.406+0000 7fc2347f8700 1 -- 172.21.15.149:0/70524577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc22c045ad0 con 0x7fc22c0f5410 2024-02-12T16:35:49.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.406+0000 7fc22affd700 1 --2- 172.21.15.149:0/70524577 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc21402ff60 0x7fc214032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:49.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.406+0000 7fc22affd700 1 -- 172.21.15.149:0/70524577 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc224041080 con 0x7fc22c0f5410 2024-02-12T16:35:49.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.406+0000 7fc231d93700 1 --2- 172.21.15.149:0/70524577 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc21402ff60 0x7fc214032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:49.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.406+0000 7fc231d93700 1 --2- 172.21.15.149:0/70524577 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc21402ff60 0x7fc214032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fc2200096a0 tx=0x7fc220006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:49.412 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.410+0000 7fc22affd700 1 -- 172.21.15.149:0/70524577 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2240095f0 con 0x7fc22c0f5410 2024-02-12T16:35:49.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.616+0000 7fc2347f8700 1 -- 172.21.15.149:0/70524577 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc22c1070b0 con 0x7fc22c0f5410 2024-02-12T16:35:49.618 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.617+0000 7fc22affd700 1 -- 172.21.15.149:0/70524577 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc22401cb10 con 0x7fc22c0f5410 2024-02-12T16:35:49.619 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:49.619 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:49.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.618+0000 7fc2347f8700 1 -- 172.21.15.149:0/70524577 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc21402ff60 msgr2=0x7fc214032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:49.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.618+0000 7fc2347f8700 1 --2- 172.21.15.149:0/70524577 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc21402ff60 0x7fc214032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fc2200096a0 tx=0x7fc220006b40).stop 2024-02-12T16:35:49.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.618+0000 7fc2347f8700 1 -- 172.21.15.149:0/70524577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c0f5410 msgr2=0x7fc22c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:49.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.618+0000 7fc2347f8700 1 --2- 172.21.15.149:0/70524577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c0f5410 0x7fc22c1056e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fc2240089f0 tx=0x7fc22400b040).stop 2024-02-12T16:35:49.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.618+0000 7fc2347f8700 1 -- 172.21.15.149:0/70524577 shutdown_connections 2024-02-12T16:35:49.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.618+0000 7fc2347f8700 1 --2- 172.21.15.149:0/70524577 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc21402ff60 0x7fc214032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:49.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.618+0000 7fc2347f8700 1 --2- 172.21.15.149:0/70524577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c0f5410 0x7fc22c1056e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:49.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.618+0000 7fc2347f8700 1 -- 172.21.15.149:0/70524577 >> 172.21.15.149:0/70524577 conn(0x7fc22c0f0b70 msgr2=0x7fc22c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:49.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.618+0000 7fc2347f8700 1 -- 172.21.15.149:0/70524577 shutdown_connections 2024-02-12T16:35:49.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:49.618+0000 7fc2347f8700 1 -- 172.21.15.149:0/70524577 wait complete. 2024-02-12T16:35:49.621 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:50.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:48.332226+0000 mgr.smithi104.ohwbhf (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:50.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:49.617972+0000 mon.smithi104 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.149:0/70524577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:51.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:51.279 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:52.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:50.332557+0000 mgr.smithi104.ohwbhf (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:53.157 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.153+0000 7fc775611700 1 -- 172.21.15.149:0/923570121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7700f5410 msgr2=0x7fc7700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:53.157 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.153+0000 7fc775611700 1 --2- 172.21.15.149:0/923570121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7700f5410 0x7fc7700f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fc764009fb0 tx=0x7fc764009b30).stop 2024-02-12T16:35:53.157 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.153+0000 7fc775611700 1 -- 172.21.15.149:0/923570121 shutdown_connections 2024-02-12T16:35:53.157 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.153+0000 7fc775611700 1 --2- 172.21.15.149:0/923570121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7700f5410 0x7fc7700f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:53.157 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.153+0000 7fc775611700 1 -- 172.21.15.149:0/923570121 >> 172.21.15.149:0/923570121 conn(0x7fc7700f0b70 msgr2=0x7fc7700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:53.157 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.153+0000 7fc775611700 1 -- 172.21.15.149:0/923570121 shutdown_connections 2024-02-12T16:35:53.157 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.153+0000 7fc775611700 1 -- 172.21.15.149:0/923570121 wait complete. 2024-02-12T16:35:53.157 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.154+0000 7fc775611700 1 Processor -- start 2024-02-12T16:35:53.157 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.154+0000 7fc775611700 1 -- start start 2024-02-12T16:35:53.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.154+0000 7fc775611700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7700f5410 0x7fc7700fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:53.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.154+0000 7fc775611700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700fb960 con 0x7fc7700f5410 2024-02-12T16:35:53.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.154+0000 7fc76effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7700f5410 0x7fc7700fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:53.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.154+0000 7fc76effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7700f5410 0x7fc7700fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:47282/0 (socket says 172.21.15.149:47282) 2024-02-12T16:35:53.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.154+0000 7fc76effd700 1 -- 172.21.15.149:0/1103443660 learned_addr learned my addr 172.21.15.149:0/1103443660 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:53.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.154+0000 7fc76effd700 1 -- 172.21.15.149:0/1103443660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc764005040 con 0x7fc7700f5410 2024-02-12T16:35:53.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.155+0000 7fc76effd700 1 --2- 172.21.15.149:0/1103443660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7700f5410 0x7fc7700fd4e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc764005010 tx=0x7fc76400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:53.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.155+0000 7fc7577fe700 1 -- 172.21.15.149:0/1103443660 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc764009320 con 0x7fc7700f5410 2024-02-12T16:35:53.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.155+0000 7fc775611700 1 -- 172.21.15.149:0/1103443660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7700fbb00 con 0x7fc7700f5410 2024-02-12T16:35:53.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.155+0000 7fc7577fe700 1 -- 172.21.15.149:0/1103443660 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc764013380 con 0x7fc7700f5410 2024-02-12T16:35:53.159 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.155+0000 7fc7577fe700 1 -- 172.21.15.149:0/1103443660 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc764014e50 con 0x7fc7700f5410 2024-02-12T16:35:53.159 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.155+0000 7fc775611700 1 -- 172.21.15.149:0/1103443660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7700fbf90 con 0x7fc7700f5410 2024-02-12T16:35:53.159 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.156+0000 7fc7577fe700 1 -- 172.21.15.149:0/1103443660 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7640095f0 con 0x7fc7700f5410 2024-02-12T16:35:53.159 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.156+0000 7fc775611700 1 -- 172.21.15.149:0/1103443660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc770059100 con 0x7fc7700f5410 2024-02-12T16:35:53.160 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.156+0000 7fc7577fe700 1 --2- 172.21.15.149:0/1103443660 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc75802ff10 0x7fc7580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:53.160 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.156+0000 7fc7577fe700 1 -- 172.21.15.149:0/1103443660 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc764040ed0 con 0x7fc7700f5410 2024-02-12T16:35:53.160 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.156+0000 7fc76e7fc700 1 --2- 172.21.15.149:0/1103443660 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc75802ff10 0x7fc7580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:53.160 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.157+0000 7fc76e7fc700 1 --2- 172.21.15.149:0/1103443660 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc75802ff10 0x7fc7580323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc7600096a0 tx=0x7fc760006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:53.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.161+0000 7fc7577fe700 1 -- 172.21.15.149:0/1103443660 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc764014420 con 0x7fc7700f5410 2024-02-12T16:35:53.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.363+0000 7fc775611700 1 -- 172.21.15.149:0/1103443660 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7700fafa0 con 0x7fc7700f5410 2024-02-12T16:35:53.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.364+0000 7fc7577fe700 1 -- 172.21.15.149:0/1103443660 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc764010230 con 0x7fc7700f5410 2024-02-12T16:35:53.367 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:53.367 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:53.368 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.366+0000 7fc775611700 1 -- 172.21.15.149:0/1103443660 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc75802ff10 msgr2=0x7fc7580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:53.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.366+0000 7fc775611700 1 --2- 172.21.15.149:0/1103443660 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc75802ff10 0x7fc7580323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc7600096a0 tx=0x7fc760006b40).stop 2024-02-12T16:35:53.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.366+0000 7fc775611700 1 -- 172.21.15.149:0/1103443660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7700f5410 msgr2=0x7fc7700fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:53.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.366+0000 7fc775611700 1 --2- 172.21.15.149:0/1103443660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7700f5410 0x7fc7700fd4e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc764005010 tx=0x7fc76400b040).stop 2024-02-12T16:35:53.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.366+0000 7fc775611700 1 -- 172.21.15.149:0/1103443660 shutdown_connections 2024-02-12T16:35:53.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.366+0000 7fc775611700 1 --2- 172.21.15.149:0/1103443660 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc75802ff10 0x7fc7580323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:53.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.366+0000 7fc775611700 1 --2- 172.21.15.149:0/1103443660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc7700f5410 0x7fc7700fd4e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:53.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.366+0000 7fc775611700 1 -- 172.21.15.149:0/1103443660 >> 172.21.15.149:0/1103443660 conn(0x7fc7700f0b70 msgr2=0x7fc7700f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:53.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.366+0000 7fc775611700 1 -- 172.21.15.149:0/1103443660 shutdown_connections 2024-02-12T16:35:53.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:53.366+0000 7fc775611700 1 -- 172.21.15.149:0/1103443660 wait complete. 2024-02-12T16:35:53.369 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:54.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:52.332823+0000 mgr.smithi104.ohwbhf (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:54.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:53.365617+0000 mon.smithi104 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.149:0/1103443660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:54.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:54.889 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:35:56.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:54.333184+0000 mgr.smithi104.ohwbhf (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:56.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.776+0000 7f5089338700 1 -- 172.21.15.149:0/1232242799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50840f59d0 msgr2=0x7f50840f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:56.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.776+0000 7f5089338700 1 --2- 172.21.15.149:0/1232242799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50840f59d0 0x7f50840f5df0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f506c009fb0 tx=0x7f506c009b30).stop 2024-02-12T16:35:56.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.776+0000 7f5089338700 1 -- 172.21.15.149:0/1232242799 shutdown_connections 2024-02-12T16:35:56.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.776+0000 7f5089338700 1 --2- 172.21.15.149:0/1232242799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50840f59d0 0x7f50840f5df0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:56.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.776+0000 7f5089338700 1 -- 172.21.15.149:0/1232242799 >> 172.21.15.149:0/1232242799 conn(0x7f50840f02e0 msgr2=0x7f50840f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:56.779 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.776+0000 7f5089338700 1 -- 172.21.15.149:0/1232242799 shutdown_connections 2024-02-12T16:35:56.780 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.777+0000 7f5089338700 1 -- 172.21.15.149:0/1232242799 wait complete. 2024-02-12T16:35:56.782 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.777+0000 7f5089338700 1 Processor -- start 2024-02-12T16:35:56.782 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.777+0000 7f5089338700 1 -- start start 2024-02-12T16:35:56.783 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f5089338700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50840f59d0 0x7f50841056d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:56.783 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f5089338700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5084105bc0 con 0x7f50840f59d0 2024-02-12T16:35:56.783 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f5082ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50840f59d0 0x7f50841056d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:56.783 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f5082ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50840f59d0 0x7f50841056d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:56932/0 (socket says 172.21.15.149:56932) 2024-02-12T16:35:56.783 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f5082ffd700 1 -- 172.21.15.149:0/3494541386 learned_addr learned my addr 172.21.15.149:0/3494541386 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:35:56.783 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f5082ffd700 1 -- 172.21.15.149:0/3494541386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c005040 con 0x7f50840f59d0 2024-02-12T16:35:56.783 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f5082ffd700 1 --2- 172.21.15.149:0/3494541386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50840f59d0 0x7f50841056d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f506c005010 tx=0x7f506c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:56.783 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f507b7fe700 1 -- 172.21.15.149:0/3494541386 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f506c009320 con 0x7f50840f59d0 2024-02-12T16:35:56.783 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f5089338700 1 -- 172.21.15.149:0/3494541386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5084105dc0 con 0x7f50840f59d0 2024-02-12T16:35:56.783 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f507b7fe700 1 -- 172.21.15.149:0/3494541386 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f506c013380 con 0x7f50840f59d0 2024-02-12T16:35:56.784 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f507b7fe700 1 -- 172.21.15.149:0/3494541386 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f506c014e50 con 0x7f50840f59d0 2024-02-12T16:35:56.784 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.778+0000 7f5089338700 1 -- 172.21.15.149:0/3494541386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5084106250 con 0x7f50840f59d0 2024-02-12T16:35:56.784 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.779+0000 7f507b7fe700 1 -- 172.21.15.149:0/3494541386 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f506c014420 con 0x7f50840f59d0 2024-02-12T16:35:56.784 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.779+0000 7f5089338700 1 -- 172.21.15.149:0/3494541386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5084045ad0 con 0x7f50840f59d0 2024-02-12T16:35:56.784 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.779+0000 7f507b7fe700 1 --2- 172.21.15.149:0/3494541386 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f507002ff10 0x7f50700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:35:56.784 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.779+0000 7f507b7fe700 1 -- 172.21.15.149:0/3494541386 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f506c040ff0 con 0x7f50840f59d0 2024-02-12T16:35:56.784 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.780+0000 7f50827fc700 1 --2- 172.21.15.149:0/3494541386 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f507002ff10 0x7f50700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:35:56.784 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.780+0000 7f50827fc700 1 --2- 172.21.15.149:0/3494541386 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f507002ff10 0x7f50700323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f50740096a0 tx=0x7f5074006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:35:56.786 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.784+0000 7f507b7fe700 1 -- 172.21.15.149:0/3494541386 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f506c06d2b0 con 0x7f50840f59d0 2024-02-12T16:35:56.991 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.989+0000 7f5089338700 1 -- 172.21.15.149:0/3494541386 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5084107100 con 0x7f50840f59d0 2024-02-12T16:35:56.991 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.989+0000 7f507b7fe700 1 -- 172.21.15.149:0/3494541386 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f506c014790 con 0x7f50840f59d0 2024-02-12T16:35:56.991 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:35:56.992 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:35:56.993 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.991+0000 7f5089338700 1 -- 172.21.15.149:0/3494541386 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f507002ff10 msgr2=0x7f50700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:56.993 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.991+0000 7f5089338700 1 --2- 172.21.15.149:0/3494541386 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f507002ff10 0x7f50700323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f50740096a0 tx=0x7f5074006b40).stop 2024-02-12T16:35:56.993 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.991+0000 7f5089338700 1 -- 172.21.15.149:0/3494541386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50840f59d0 msgr2=0x7f50841056d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:35:56.993 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.991+0000 7f5089338700 1 --2- 172.21.15.149:0/3494541386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50840f59d0 0x7f50841056d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f506c005010 tx=0x7f506c00b040).stop 2024-02-12T16:35:56.993 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.991+0000 7f5089338700 1 -- 172.21.15.149:0/3494541386 shutdown_connections 2024-02-12T16:35:56.993 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.991+0000 7f5089338700 1 --2- 172.21.15.149:0/3494541386 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f507002ff10 0x7f50700323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:56.993 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.991+0000 7f5089338700 1 --2- 172.21.15.149:0/3494541386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50840f59d0 0x7f50841056d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:35:56.993 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.991+0000 7f5089338700 1 -- 172.21.15.149:0/3494541386 >> 172.21.15.149:0/3494541386 conn(0x7f50840f02e0 msgr2=0x7f50840f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:35:56.994 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.991+0000 7f5089338700 1 -- 172.21.15.149:0/3494541386 shutdown_connections 2024-02-12T16:35:56.994 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:35:56.991+0000 7f5089338700 1 -- 172.21.15.149:0/3494541386 wait complete. 2024-02-12T16:35:56.994 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:35:57.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:57 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:35:56.990679+0000 mon.smithi104 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.149:0/3494541386' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:35:58.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:35:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:56.333561+0000 mgr.smithi104.ohwbhf (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:35:58.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:35:58.747 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:00.482 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.480+0000 7f695d703700 1 -- 172.21.15.149:0/3581145609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69580f5410 msgr2=0x7f69580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:00.482 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.480+0000 7f695d703700 1 --2- 172.21.15.149:0/3581145609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69580f5410 0x7f69580f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f694c009fb0 tx=0x7f694c009b30).stop 2024-02-12T16:36:00.482 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.480+0000 7f695d703700 1 -- 172.21.15.149:0/3581145609 shutdown_connections 2024-02-12T16:36:00.482 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.480+0000 7f695d703700 1 --2- 172.21.15.149:0/3581145609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69580f5410 0x7f69580f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:00.482 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.480+0000 7f695d703700 1 -- 172.21.15.149:0/3581145609 >> 172.21.15.149:0/3581145609 conn(0x7f69580f0b70 msgr2=0x7f69580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:00.482 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.480+0000 7f695d703700 1 -- 172.21.15.149:0/3581145609 shutdown_connections 2024-02-12T16:36:00.482 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.480+0000 7f695d703700 1 -- 172.21.15.149:0/3581145609 wait complete. 2024-02-12T16:36:00.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.481+0000 7f695d703700 1 Processor -- start 2024-02-12T16:36:00.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.481+0000 7f695d703700 1 -- start start 2024-02-12T16:36:00.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.481+0000 7f695d703700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69580f5410 0x7f6958105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:00.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.481+0000 7f695d703700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6958105c10 con 0x7f69580f5410 2024-02-12T16:36:00.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.482+0000 7f6956ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69580f5410 0x7f6958105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.482+0000 7f6956ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69580f5410 0x7f6958105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:56954/0 (socket says 172.21.15.149:56954) 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.482+0000 7f6956ffd700 1 -- 172.21.15.149:0/439295365 learned_addr learned my addr 172.21.15.149:0/439295365 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.482+0000 7f6956ffd700 1 -- 172.21.15.149:0/439295365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c005040 con 0x7f69580f5410 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.482+0000 7f6956ffd700 1 --2- 172.21.15.149:0/439295365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69580f5410 0x7f6958105720 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f694c005010 tx=0x7f694c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.482+0000 7f693f7fe700 1 -- 172.21.15.149:0/439295365 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f694c009320 con 0x7f69580f5410 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.482+0000 7f693f7fe700 1 -- 172.21.15.149:0/439295365 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f694c013380 con 0x7f69580f5410 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.482+0000 7f695d703700 1 -- 172.21.15.149:0/439295365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6958105e10 con 0x7f69580f5410 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.482+0000 7f693f7fe700 1 -- 172.21.15.149:0/439295365 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f694c014e50 con 0x7f69580f5410 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.482+0000 7f695d703700 1 -- 172.21.15.149:0/439295365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69581062a0 con 0x7f69580f5410 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.483+0000 7f693f7fe700 1 -- 172.21.15.149:0/439295365 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f694c014420 con 0x7f69580f5410 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.483+0000 7f695d703700 1 -- 172.21.15.149:0/439295365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6958045ad0 con 0x7f69580f5410 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.483+0000 7f693f7fe700 1 --2- 172.21.15.149:0/439295365 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f694002ff10 0x7f69400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.483+0000 7f693f7fe700 1 -- 172.21.15.149:0/439295365 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f694c040f70 con 0x7f69580f5410 2024-02-12T16:36:00.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.483+0000 7f69567fc700 1 --2- 172.21.15.149:0/439295365 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f694002ff10 0x7f69400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:00.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.484+0000 7f69567fc700 1 --2- 172.21.15.149:0/439295365 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f694002ff10 0x7f69400323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f69480096a0 tx=0x7f6948006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:00.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.488+0000 7f693f7fe700 1 -- 172.21.15.149:0/439295365 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f694c0095f0 con 0x7f69580f5410 2024-02-12T16:36:00.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:35:58.333946+0000 mgr.smithi104.ohwbhf (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:00.693 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.692+0000 7f695d703700 1 -- 172.21.15.149:0/439295365 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6958107150 con 0x7f69580f5410 2024-02-12T16:36:00.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.693+0000 7f693f7fe700 1 -- 172.21.15.149:0/439295365 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f694c01db70 con 0x7f69580f5410 2024-02-12T16:36:00.694 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:00.695 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:00.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.695+0000 7f695d703700 1 -- 172.21.15.149:0/439295365 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f694002ff10 msgr2=0x7f69400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:00.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.695+0000 7f695d703700 1 --2- 172.21.15.149:0/439295365 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f694002ff10 0x7f69400323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f69480096a0 tx=0x7f6948006b40).stop 2024-02-12T16:36:00.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.695+0000 7f695d703700 1 -- 172.21.15.149:0/439295365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69580f5410 msgr2=0x7f6958105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:00.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.695+0000 7f695d703700 1 --2- 172.21.15.149:0/439295365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69580f5410 0x7f6958105720 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f694c005010 tx=0x7f694c00b040).stop 2024-02-12T16:36:00.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.695+0000 7f695d703700 1 -- 172.21.15.149:0/439295365 shutdown_connections 2024-02-12T16:36:00.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.695+0000 7f695d703700 1 --2- 172.21.15.149:0/439295365 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f694002ff10 0x7f69400323d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:00.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.695+0000 7f695d703700 1 --2- 172.21.15.149:0/439295365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69580f5410 0x7f6958105720 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:00.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.695+0000 7f695d703700 1 -- 172.21.15.149:0/439295365 >> 172.21.15.149:0/439295365 conn(0x7f69580f0b70 msgr2=0x7f69580f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:00.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.695+0000 7f695d703700 1 -- 172.21.15.149:0/439295365 shutdown_connections 2024-02-12T16:36:00.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:00.695+0000 7f695d703700 1 -- 172.21.15.149:0/439295365 wait complete. 2024-02-12T16:36:00.699 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:01.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:00.693612+0000 mon.smithi104 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.149:0/439295365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:02.333 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:02.333 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:02.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:00.334353+0000 mgr.smithi104.ohwbhf (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:04.242 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.240+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/2352801190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c00f5410 msgr2=0x7ff6c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:04.242 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.240+0000 7ff6c4fe2700 1 --2- 172.21.15.149:0/2352801190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c00f5410 0x7ff6c00f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff6a8009fb0 tx=0x7ff6a8009b30).stop 2024-02-12T16:36:04.242 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.240+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/2352801190 shutdown_connections 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.240+0000 7ff6c4fe2700 1 --2- 172.21.15.149:0/2352801190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c00f5410 0x7ff6c00f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.240+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/2352801190 >> 172.21.15.149:0/2352801190 conn(0x7ff6c00f0b70 msgr2=0x7ff6c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.240+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/2352801190 shutdown_connections 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.240+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/2352801190 wait complete. 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.240+0000 7ff6c4fe2700 1 Processor -- start 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.240+0000 7ff6c4fe2700 1 -- start start 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.240+0000 7ff6c4fe2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c00f5410 0x7ff6c01067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.240+0000 7ff6c4fe2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c0106ca0 con 0x7ff6c00f5410 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.241+0000 7ff6be59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c00f5410 0x7ff6c01067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.241+0000 7ff6be59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c00f5410 0x7ff6c01067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:56970/0 (socket says 172.21.15.149:56970) 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.241+0000 7ff6be59c700 1 -- 172.21.15.149:0/3548358276 learned_addr learned my addr 172.21.15.149:0/3548358276 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.241+0000 7ff6be59c700 1 -- 172.21.15.149:0/3548358276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a8005040 con 0x7ff6c00f5410 2024-02-12T16:36:04.243 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.241+0000 7ff6be59c700 1 --2- 172.21.15.149:0/3548358276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c00f5410 0x7ff6c01067b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7ff6a8005010 tx=0x7ff6a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:04.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.241+0000 7ff6b6ffd700 1 -- 172.21.15.149:0/3548358276 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6a8009320 con 0x7ff6c00f5410 2024-02-12T16:36:04.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.241+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/3548358276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c0106ea0 con 0x7ff6c00f5410 2024-02-12T16:36:04.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.241+0000 7ff6b6ffd700 1 -- 172.21.15.149:0/3548358276 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6a8013380 con 0x7ff6c00f5410 2024-02-12T16:36:04.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.241+0000 7ff6b6ffd700 1 -- 172.21.15.149:0/3548358276 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6a8014e50 con 0x7ff6c00f5410 2024-02-12T16:36:04.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.241+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/3548358276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c0107330 con 0x7ff6c00f5410 2024-02-12T16:36:04.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.242+0000 7ff6b6ffd700 1 -- 172.21.15.149:0/3548358276 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6a80095f0 con 0x7ff6c00f5410 2024-02-12T16:36:04.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.242+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/3548358276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c0045ad0 con 0x7ff6c00f5410 2024-02-12T16:36:04.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.242+0000 7ff6b6ffd700 1 --2- 172.21.15.149:0/3548358276 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff6ac02ff10 0x7ff6ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:04.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.242+0000 7ff6b6ffd700 1 -- 172.21.15.149:0/3548358276 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6a8040fa0 con 0x7ff6c00f5410 2024-02-12T16:36:04.245 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.242+0000 7ff6bdd9b700 1 --2- 172.21.15.149:0/3548358276 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff6ac02ff10 0x7ff6ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:04.246 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.243+0000 7ff6bdd9b700 1 --2- 172.21.15.149:0/3548358276 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff6ac02ff10 0x7ff6ac0323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff6b00096a0 tx=0x7ff6b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:04.248 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.247+0000 7ff6b6ffd700 1 -- 172.21.15.149:0/3548358276 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6a8014420 con 0x7ff6c00f5410 2024-02-12T16:36:04.452 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.451+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/3548358276 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6c00fafa0 con 0x7ff6c00f5410 2024-02-12T16:36:04.453 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.452+0000 7ff6b6ffd700 1 -- 172.21.15.149:0/3548358276 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff6a8010230 con 0x7ff6c00f5410 2024-02-12T16:36:04.453 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:04.454 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:04.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.454+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/3548358276 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff6ac02ff10 msgr2=0x7ff6ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:04.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.454+0000 7ff6c4fe2700 1 --2- 172.21.15.149:0/3548358276 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff6ac02ff10 0x7ff6ac0323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff6b00096a0 tx=0x7ff6b0006b40).stop 2024-02-12T16:36:04.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.454+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/3548358276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c00f5410 msgr2=0x7ff6c01067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:04.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.454+0000 7ff6c4fe2700 1 --2- 172.21.15.149:0/3548358276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c00f5410 0x7ff6c01067b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7ff6a8005010 tx=0x7ff6a800b040).stop 2024-02-12T16:36:04.456 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.454+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/3548358276 shutdown_connections 2024-02-12T16:36:04.456 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.454+0000 7ff6c4fe2700 1 --2- 172.21.15.149:0/3548358276 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff6ac02ff10 0x7ff6ac0323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:04.456 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.454+0000 7ff6c4fe2700 1 --2- 172.21.15.149:0/3548358276 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6c00f5410 0x7ff6c01067b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:04.456 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.454+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/3548358276 >> 172.21.15.149:0/3548358276 conn(0x7ff6c00f0b70 msgr2=0x7ff6c005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:04.456 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.454+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/3548358276 shutdown_connections 2024-02-12T16:36:04.456 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:04.454+0000 7ff6c4fe2700 1 -- 172.21.15.149:0/3548358276 wait complete. 2024-02-12T16:36:04.456 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:04.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:02.334661+0000 mgr.smithi104.ohwbhf (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:05.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:04.452406+0000 mon.smithi104 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.149:0/3548358276' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:06.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:06.215 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:06.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:04.334998+0000 mgr.smithi104.ohwbhf (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:08.130 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.126+0000 7fe0461d9700 1 -- 172.21.15.149:0/3763672635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe040060200 msgr2=0x7fe0400fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:08.130 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.126+0000 7fe0461d9700 1 --2- 172.21.15.149:0/3763672635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe040060200 0x7fe0400fb570 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fe028009fb0 tx=0x7fe028009b30).stop 2024-02-12T16:36:08.131 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.126+0000 7fe0461d9700 1 -- 172.21.15.149:0/3763672635 shutdown_connections 2024-02-12T16:36:08.131 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.126+0000 7fe0461d9700 1 --2- 172.21.15.149:0/3763672635 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe040060200 0x7fe0400fb570 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:08.131 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.126+0000 7fe0461d9700 1 -- 172.21.15.149:0/3763672635 >> 172.21.15.149:0/3763672635 conn(0x7fe0400f0b70 msgr2=0x7fe0400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:08.131 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.126+0000 7fe0461d9700 1 -- 172.21.15.149:0/3763672635 shutdown_connections 2024-02-12T16:36:08.131 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.126+0000 7fe0461d9700 1 -- 172.21.15.149:0/3763672635 wait complete. 2024-02-12T16:36:08.131 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.127+0000 7fe0461d9700 1 Processor -- start 2024-02-12T16:36:08.131 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.127+0000 7fe0461d9700 1 -- start start 2024-02-12T16:36:08.131 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.127+0000 7fe0461d9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe040060200 0x7fe0401089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:08.131 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.127+0000 7fe0461d9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe040108e90 con 0x7fe040060200 2024-02-12T16:36:08.132 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.127+0000 7fe03f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe040060200 0x7fe0401089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:08.132 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.127+0000 7fe03f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe040060200 0x7fe0401089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:40372/0 (socket says 172.21.15.149:40372) 2024-02-12T16:36:08.132 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.127+0000 7fe03f7fe700 1 -- 172.21.15.149:0/3032869481 learned_addr learned my addr 172.21.15.149:0/3032869481 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:08.132 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.127+0000 7fe03f7fe700 1 -- 172.21.15.149:0/3032869481 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe028005040 con 0x7fe040060200 2024-02-12T16:36:08.132 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.128+0000 7fe03f7fe700 1 --2- 172.21.15.149:0/3032869481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe040060200 0x7fe0401089a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fe028009830 tx=0x7fe02800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:08.132 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.128+0000 7fe037fff700 1 -- 172.21.15.149:0/3032869481 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe028009320 con 0x7fe040060200 2024-02-12T16:36:08.132 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.128+0000 7fe0461d9700 1 -- 172.21.15.149:0/3032869481 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe040109090 con 0x7fe040060200 2024-02-12T16:36:08.132 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.128+0000 7fe037fff700 1 -- 172.21.15.149:0/3032869481 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe028013380 con 0x7fe040060200 2024-02-12T16:36:08.132 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.128+0000 7fe037fff700 1 -- 172.21.15.149:0/3032869481 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe028014420 con 0x7fe040060200 2024-02-12T16:36:08.132 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.128+0000 7fe0461d9700 1 -- 172.21.15.149:0/3032869481 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe040109520 con 0x7fe040060200 2024-02-12T16:36:08.133 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.129+0000 7fe037fff700 1 -- 172.21.15.149:0/3032869481 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe02801e420 con 0x7fe040060200 2024-02-12T16:36:08.133 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.129+0000 7fe0461d9700 1 -- 172.21.15.149:0/3032869481 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe040045ad0 con 0x7fe040060200 2024-02-12T16:36:08.134 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.129+0000 7fe037fff700 1 --2- 172.21.15.149:0/3032869481 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe02c02ff60 0x7fe02c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:08.134 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.129+0000 7fe037fff700 1 -- 172.21.15.149:0/3032869481 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe028040e20 con 0x7fe040060200 2024-02-12T16:36:08.134 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.129+0000 7fe03effd700 1 --2- 172.21.15.149:0/3032869481 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe02c02ff60 0x7fe02c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:08.134 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.130+0000 7fe03effd700 1 --2- 172.21.15.149:0/3032869481 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe02c02ff60 0x7fe02c032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe0300096a0 tx=0x7fe030006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:08.135 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.134+0000 7fe037fff700 1 -- 172.21.15.149:0/3032869481 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe02801c970 con 0x7fe040060200 2024-02-12T16:36:08.338 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.336+0000 7fe0461d9700 1 -- 172.21.15.149:0/3032869481 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0400fe290 con 0x7fe040060200 2024-02-12T16:36:08.338 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.336+0000 7fe037fff700 1 -- 172.21.15.149:0/3032869481 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe02802c2d0 con 0x7fe040060200 2024-02-12T16:36:08.338 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:08.338 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:08.340 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.338+0000 7fe0461d9700 1 -- 172.21.15.149:0/3032869481 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe02c02ff60 msgr2=0x7fe02c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:08.340 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.338+0000 7fe0461d9700 1 --2- 172.21.15.149:0/3032869481 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe02c02ff60 0x7fe02c032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe0300096a0 tx=0x7fe030006b40).stop 2024-02-12T16:36:08.340 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.338+0000 7fe0461d9700 1 -- 172.21.15.149:0/3032869481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe040060200 msgr2=0x7fe0401089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:08.340 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.338+0000 7fe0461d9700 1 --2- 172.21.15.149:0/3032869481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe040060200 0x7fe0401089a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fe028009830 tx=0x7fe02800b040).stop 2024-02-12T16:36:08.340 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.338+0000 7fe0461d9700 1 -- 172.21.15.149:0/3032869481 shutdown_connections 2024-02-12T16:36:08.340 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.338+0000 7fe0461d9700 1 --2- 172.21.15.149:0/3032869481 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe02c02ff60 0x7fe02c032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:08.340 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.338+0000 7fe0461d9700 1 --2- 172.21.15.149:0/3032869481 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe040060200 0x7fe0401089a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:08.340 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.338+0000 7fe0461d9700 1 -- 172.21.15.149:0/3032869481 >> 172.21.15.149:0/3032869481 conn(0x7fe0400f0b70 msgr2=0x7fe0400f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:08.340 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.338+0000 7fe0461d9700 1 -- 172.21.15.149:0/3032869481 shutdown_connections 2024-02-12T16:36:08.340 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:08.338+0000 7fe0461d9700 1 -- 172.21.15.149:0/3032869481 wait complete. 2024-02-12T16:36:08.341 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:08.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:06.335348+0000 mgr.smithi104.ohwbhf (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:09.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:08.337655+0000 mon.smithi104 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.149:0/3032869481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:09.989 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:09.989 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:10.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:08.335681+0000 mgr.smithi104.ohwbhf (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:11.843 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.840+0000 7f0fa67cd700 1 -- 172.21.15.149:0/1479837729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fa00f5410 msgr2=0x7f0fa00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:11.843 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.840+0000 7f0fa67cd700 1 --2- 172.21.15.149:0/1479837729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fa00f5410 0x7f0fa00f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f0f90009fb0 tx=0x7f0f90009b30).stop 2024-02-12T16:36:11.843 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.840+0000 7f0fa67cd700 1 -- 172.21.15.149:0/1479837729 shutdown_connections 2024-02-12T16:36:11.843 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.840+0000 7f0fa67cd700 1 --2- 172.21.15.149:0/1479837729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fa00f5410 0x7f0fa00f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:11.843 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.840+0000 7f0fa67cd700 1 -- 172.21.15.149:0/1479837729 >> 172.21.15.149:0/1479837729 conn(0x7f0fa00f0b70 msgr2=0x7f0fa00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:11.843 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.840+0000 7f0fa67cd700 1 -- 172.21.15.149:0/1479837729 shutdown_connections 2024-02-12T16:36:11.843 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.840+0000 7f0fa67cd700 1 -- 172.21.15.149:0/1479837729 wait complete. 2024-02-12T16:36:11.843 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.841+0000 7f0fa67cd700 1 Processor -- start 2024-02-12T16:36:11.845 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.841+0000 7f0fa67cd700 1 -- start start 2024-02-12T16:36:11.845 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.841+0000 7f0fa67cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fa00f5410 0x7f0fa00fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:11.845 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.841+0000 7f0fa67cd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00fb960 con 0x7f0fa00f5410 2024-02-12T16:36:11.845 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.841+0000 7f0f9ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fa00f5410 0x7f0fa00fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:11.845 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.841+0000 7f0f9ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fa00f5410 0x7f0fa00fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:40394/0 (socket says 172.21.15.149:40394) 2024-02-12T16:36:11.845 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.841+0000 7f0f9ffff700 1 -- 172.21.15.149:0/3663076091 learned_addr learned my addr 172.21.15.149:0/3663076091 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:11.845 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.842+0000 7f0f9ffff700 1 -- 172.21.15.149:0/3663076091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f90005040 con 0x7f0fa00f5410 2024-02-12T16:36:11.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.842+0000 7f0f9ffff700 1 --2- 172.21.15.149:0/3663076091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fa00f5410 0x7f0fa00fd4e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f0f90005010 tx=0x7f0f9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:11.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.842+0000 7f0f9cff9700 1 -- 172.21.15.149:0/3663076091 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f90009320 con 0x7f0fa00f5410 2024-02-12T16:36:11.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.842+0000 7f0fa67cd700 1 -- 172.21.15.149:0/3663076091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa00fbb00 con 0x7f0fa00f5410 2024-02-12T16:36:11.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.842+0000 7f0f9cff9700 1 -- 172.21.15.149:0/3663076091 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f90013380 con 0x7f0fa00f5410 2024-02-12T16:36:11.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.842+0000 7f0f9cff9700 1 -- 172.21.15.149:0/3663076091 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f90014e50 con 0x7f0fa00f5410 2024-02-12T16:36:11.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.842+0000 7f0fa67cd700 1 -- 172.21.15.149:0/3663076091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa00fbf90 con 0x7f0fa00f5410 2024-02-12T16:36:11.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.843+0000 7f0f9cff9700 1 -- 172.21.15.149:0/3663076091 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f900095f0 con 0x7f0fa00f5410 2024-02-12T16:36:11.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.843+0000 7f0fa67cd700 1 -- 172.21.15.149:0/3663076091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa0059100 con 0x7f0fa00f5410 2024-02-12T16:36:11.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.843+0000 7f0f9cff9700 1 --2- 172.21.15.149:0/3663076091 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0f8802ff10 0x7f0f880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:11.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.843+0000 7f0f9cff9700 1 -- 172.21.15.149:0/3663076091 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f90040f70 con 0x7f0fa00f5410 2024-02-12T16:36:11.846 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.843+0000 7f0f9f7fe700 1 --2- 172.21.15.149:0/3663076091 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0f8802ff10 0x7f0f880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:11.847 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.844+0000 7f0f9f7fe700 1 --2- 172.21.15.149:0/3663076091 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0f8802ff10 0x7f0f880323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f0f940096a0 tx=0x7f0f94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:11.850 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:11.848+0000 7f0f9cff9700 1 -- 172.21.15.149:0/3663076091 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f90014420 con 0x7f0fa00f5410 2024-02-12T16:36:12.052 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.050+0000 7f0fa67cd700 1 -- 172.21.15.149:0/3663076091 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fa0045ad0 con 0x7f0fa00f5410 2024-02-12T16:36:12.054 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.050+0000 7f0f9cff9700 1 -- 172.21.15.149:0/3663076091 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0f90010230 con 0x7f0fa00f5410 2024-02-12T16:36:12.054 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:12.054 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:12.055 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.052+0000 7f0fa67cd700 1 -- 172.21.15.149:0/3663076091 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0f8802ff10 msgr2=0x7f0f880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:12.055 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.052+0000 7f0fa67cd700 1 --2- 172.21.15.149:0/3663076091 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0f8802ff10 0x7f0f880323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f0f940096a0 tx=0x7f0f94006b40).stop 2024-02-12T16:36:12.055 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.052+0000 7f0fa67cd700 1 -- 172.21.15.149:0/3663076091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fa00f5410 msgr2=0x7f0fa00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:12.055 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.052+0000 7f0fa67cd700 1 --2- 172.21.15.149:0/3663076091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fa00f5410 0x7f0fa00fd4e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f0f90005010 tx=0x7f0f9000b040).stop 2024-02-12T16:36:12.056 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.052+0000 7f0fa67cd700 1 -- 172.21.15.149:0/3663076091 shutdown_connections 2024-02-12T16:36:12.056 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.052+0000 7f0fa67cd700 1 --2- 172.21.15.149:0/3663076091 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0f8802ff10 0x7f0f880323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:12.056 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.052+0000 7f0fa67cd700 1 --2- 172.21.15.149:0/3663076091 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0fa00f5410 0x7f0fa00fd4e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:12.056 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.052+0000 7f0fa67cd700 1 -- 172.21.15.149:0/3663076091 >> 172.21.15.149:0/3663076091 conn(0x7f0fa00f0b70 msgr2=0x7f0fa005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:12.056 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.052+0000 7f0fa67cd700 1 -- 172.21.15.149:0/3663076091 shutdown_connections 2024-02-12T16:36:12.056 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:12.052+0000 7f0fa67cd700 1 -- 172.21.15.149:0/3663076091 wait complete. 2024-02-12T16:36:12.056 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:12.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:10.336036+0000 mgr.smithi104.ohwbhf (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:12.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:12.051643+0000 mon.smithi104 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.149:0/3663076091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:13.861 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:13.861 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:14.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:12.336397+0000 mgr.smithi104.ohwbhf (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:15.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.673+0000 7fb7a9904700 1 -- 172.21.15.149:0/3283956624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a40f5410 msgr2=0x7fb7a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:15.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.673+0000 7fb7a9904700 1 --2- 172.21.15.149:0/3283956624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a40f5410 0x7fb7a40f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fb798009fb0 tx=0x7fb798009b30).stop 2024-02-12T16:36:15.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.673+0000 7fb7a9904700 1 -- 172.21.15.149:0/3283956624 shutdown_connections 2024-02-12T16:36:15.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.673+0000 7fb7a9904700 1 --2- 172.21.15.149:0/3283956624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a40f5410 0x7fb7a40f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:15.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.673+0000 7fb7a9904700 1 -- 172.21.15.149:0/3283956624 >> 172.21.15.149:0/3283956624 conn(0x7fb7a40f0b70 msgr2=0x7fb7a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:15.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.673+0000 7fb7a9904700 1 -- 172.21.15.149:0/3283956624 shutdown_connections 2024-02-12T16:36:15.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.673+0000 7fb7a9904700 1 -- 172.21.15.149:0/3283956624 wait complete. 2024-02-12T16:36:15.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.674+0000 7fb7a9904700 1 Processor -- start 2024-02-12T16:36:15.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.674+0000 7fb7a9904700 1 -- start start 2024-02-12T16:36:15.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.674+0000 7fb7a9904700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a40f5410 0x7fb7a4109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:15.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.674+0000 7fb7a9904700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4060220 con 0x7fb7a40f5410 2024-02-12T16:36:15.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.674+0000 7fb7a2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a40f5410 0x7fb7a4109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:15.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.674+0000 7fb7a2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a40f5410 0x7fb7a4109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55428/0 (socket says 172.21.15.149:55428) 2024-02-12T16:36:15.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.674+0000 7fb7a2ffd700 1 -- 172.21.15.149:0/4047512162 learned_addr learned my addr 172.21.15.149:0/4047512162 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:15.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.674+0000 7fb7a2ffd700 1 -- 172.21.15.149:0/4047512162 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb798005040 con 0x7fb7a40f5410 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.674+0000 7fb7a2ffd700 1 --2- 172.21.15.149:0/4047512162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a40f5410 0x7fb7a4109500 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fb798005010 tx=0x7fb79800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.675+0000 7fb78b7fe700 1 -- 172.21.15.149:0/4047512162 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb798009320 con 0x7fb7a40f5410 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.675+0000 7fb78b7fe700 1 -- 172.21.15.149:0/4047512162 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb798013380 con 0x7fb7a40f5410 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.675+0000 7fb7a9904700 1 -- 172.21.15.149:0/4047512162 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a4060420 con 0x7fb7a40f5410 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.675+0000 7fb78b7fe700 1 -- 172.21.15.149:0/4047512162 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb798014e50 con 0x7fb7a40f5410 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.675+0000 7fb7a9904700 1 -- 172.21.15.149:0/4047512162 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a4109bb0 con 0x7fb7a40f5410 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.676+0000 7fb78b7fe700 1 -- 172.21.15.149:0/4047512162 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7980095f0 con 0x7fb7a40f5410 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.676+0000 7fb78b7fe700 1 --2- 172.21.15.149:0/4047512162 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb78c02ff10 0x7fb78c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.676+0000 7fb78b7fe700 1 -- 172.21.15.149:0/4047512162 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb798040e90 con 0x7fb7a40f5410 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.676+0000 7fb7a9904700 1 -- 172.21.15.149:0/4047512162 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a4059100 con 0x7fb7a40f5410 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.676+0000 7fb7a27fc700 1 --2- 172.21.15.149:0/4047512162 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb78c02ff10 0x7fb78c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:15.679 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.676+0000 7fb7a27fc700 1 --2- 172.21.15.149:0/4047512162 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb78c02ff10 0x7fb78c0323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fb7940096a0 tx=0x7fb794006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:15.682 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.681+0000 7fb78b7fe700 1 -- 172.21.15.149:0/4047512162 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb798014420 con 0x7fb7a40f5410 2024-02-12T16:36:15.886 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.884+0000 7fb7a9904700 1 -- 172.21.15.149:0/4047512162 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7a4045ad0 con 0x7fb7a40f5410 2024-02-12T16:36:15.887 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.885+0000 7fb78b7fe700 1 -- 172.21.15.149:0/4047512162 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb798010230 con 0x7fb7a40f5410 2024-02-12T16:36:15.887 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:15.887 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.887+0000 7fb7a9904700 1 -- 172.21.15.149:0/4047512162 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb78c02ff10 msgr2=0x7fb78c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.887+0000 7fb7a9904700 1 --2- 172.21.15.149:0/4047512162 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb78c02ff10 0x7fb78c0323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fb7940096a0 tx=0x7fb794006b40).stop 2024-02-12T16:36:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.887+0000 7fb7a9904700 1 -- 172.21.15.149:0/4047512162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a40f5410 msgr2=0x7fb7a4109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.887+0000 7fb7a9904700 1 --2- 172.21.15.149:0/4047512162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a40f5410 0x7fb7a4109500 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fb798005010 tx=0x7fb79800b040).stop 2024-02-12T16:36:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.887+0000 7fb7a9904700 1 -- 172.21.15.149:0/4047512162 shutdown_connections 2024-02-12T16:36:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.887+0000 7fb7a9904700 1 --2- 172.21.15.149:0/4047512162 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb78c02ff10 0x7fb78c0323d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.887+0000 7fb7a9904700 1 --2- 172.21.15.149:0/4047512162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7a40f5410 0x7fb7a4109500 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:15.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.887+0000 7fb7a9904700 1 -- 172.21.15.149:0/4047512162 >> 172.21.15.149:0/4047512162 conn(0x7fb7a40f0b70 msgr2=0x7fb7a40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:15.890 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.887+0000 7fb7a9904700 1 -- 172.21.15.149:0/4047512162 shutdown_connections 2024-02-12T16:36:15.890 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:15.887+0000 7fb7a9904700 1 -- 172.21.15.149:0/4047512162 wait complete. 2024-02-12T16:36:15.890 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:16.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:14.336777+0000 mgr.smithi104.ohwbhf (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:16.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:15.886398+0000 mon.smithi104 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.149:0/4047512162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:17.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:17.623 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:18.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:16.337083+0000 mgr.smithi104.ohwbhf (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:19.471 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.461+0000 7f719cca3700 1 -- 172.21.15.149:0/2604659185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71980f6510 msgr2=0x7f71980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:19.471 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.461+0000 7f719cca3700 1 --2- 172.21.15.149:0/2604659185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71980f6510 0x7f71980f6930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f718c009fb0 tx=0x7f718c009b30).stop 2024-02-12T16:36:19.472 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.461+0000 7f719cca3700 1 -- 172.21.15.149:0/2604659185 shutdown_connections 2024-02-12T16:36:19.472 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.461+0000 7f719cca3700 1 --2- 172.21.15.149:0/2604659185 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71980f6510 0x7f71980f6930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:19.472 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.461+0000 7f719cca3700 1 -- 172.21.15.149:0/2604659185 >> 172.21.15.149:0/2604659185 conn(0x7f71980f1c70 msgr2=0x7f71980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:19.472 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.461+0000 7f719cca3700 1 -- 172.21.15.149:0/2604659185 shutdown_connections 2024-02-12T16:36:19.472 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.461+0000 7f719cca3700 1 -- 172.21.15.149:0/2604659185 wait complete. 2024-02-12T16:36:19.472 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.461+0000 7f719cca3700 1 Processor -- start 2024-02-12T16:36:19.472 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.461+0000 7f719cca3700 1 -- start start 2024-02-12T16:36:19.472 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f719cca3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71980f6510 0x7f719806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:19.472 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f719cca3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7198069e20 con 0x7f71980f6510 2024-02-12T16:36:19.472 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f719659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71980f6510 0x7f719806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:19.472 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f719659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71980f6510 0x7f719806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55446/0 (socket says 172.21.15.149:55446) 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f719659c700 1 -- 172.21.15.149:0/2296168109 learned_addr learned my addr 172.21.15.149:0/2296168109 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f719659c700 1 -- 172.21.15.149:0/2296168109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718c005040 con 0x7f71980f6510 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f719659c700 1 --2- 172.21.15.149:0/2296168109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71980f6510 0x7f719806b9a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f718c005010 tx=0x7f718c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f7186ffd700 1 -- 172.21.15.149:0/2296168109 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f718c009320 con 0x7f71980f6510 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f719cca3700 1 -- 172.21.15.149:0/2296168109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7198069fc0 con 0x7f71980f6510 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f7186ffd700 1 -- 172.21.15.149:0/2296168109 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f718c013380 con 0x7f71980f6510 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f7186ffd700 1 -- 172.21.15.149:0/2296168109 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f718c014e50 con 0x7f71980f6510 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.462+0000 7f719cca3700 1 -- 172.21.15.149:0/2296168109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719806a450 con 0x7f71980f6510 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.463+0000 7f7186ffd700 1 -- 172.21.15.149:0/2296168109 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f718c0263a0 con 0x7f71980f6510 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.463+0000 7f7186ffd700 1 --2- 172.21.15.149:0/2296168109 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f718002ff10 0x7f71800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.463+0000 7f7186ffd700 1 -- 172.21.15.149:0/2296168109 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f718c040fb0 con 0x7f71980f6510 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.463+0000 7f719cca3700 1 -- 172.21.15.149:0/2296168109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7198059100 con 0x7f71980f6510 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.464+0000 7f7195d9b700 1 --2- 172.21.15.149:0/2296168109 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f718002ff10 0x7f71800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:19.473 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.464+0000 7f7195d9b700 1 --2- 172.21.15.149:0/2296168109 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f718002ff10 0x7f71800323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f71880096a0 tx=0x7f7188006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:19.474 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.468+0000 7f7186ffd700 1 -- 172.21.15.149:0/2296168109 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f718c012920 con 0x7f71980f6510 2024-02-12T16:36:19.673 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.671+0000 7f719cca3700 1 -- 172.21.15.149:0/2296168109 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f719806ab50 con 0x7f71980f6510 2024-02-12T16:36:19.674 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.672+0000 7f7186ffd700 1 -- 172.21.15.149:0/2296168109 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f718c012330 con 0x7f71980f6510 2024-02-12T16:36:19.675 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:19.675 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:19.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.674+0000 7f719cca3700 1 -- 172.21.15.149:0/2296168109 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f718002ff10 msgr2=0x7f71800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:19.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.674+0000 7f719cca3700 1 --2- 172.21.15.149:0/2296168109 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f718002ff10 0x7f71800323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f71880096a0 tx=0x7f7188006b40).stop 2024-02-12T16:36:19.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.674+0000 7f719cca3700 1 -- 172.21.15.149:0/2296168109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71980f6510 msgr2=0x7f719806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:19.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.674+0000 7f719cca3700 1 --2- 172.21.15.149:0/2296168109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71980f6510 0x7f719806b9a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f718c005010 tx=0x7f718c00b040).stop 2024-02-12T16:36:19.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.674+0000 7f719cca3700 1 -- 172.21.15.149:0/2296168109 shutdown_connections 2024-02-12T16:36:19.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.674+0000 7f719cca3700 1 --2- 172.21.15.149:0/2296168109 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f718002ff10 0x7f71800323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:19.677 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.674+0000 7f719cca3700 1 --2- 172.21.15.149:0/2296168109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71980f6510 0x7f719806b9a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:19.677 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.674+0000 7f719cca3700 1 -- 172.21.15.149:0/2296168109 >> 172.21.15.149:0/2296168109 conn(0x7f71980f1c70 msgr2=0x7f71980f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:19.677 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.674+0000 7f719cca3700 1 -- 172.21.15.149:0/2296168109 shutdown_connections 2024-02-12T16:36:19.677 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:19.674+0000 7f719cca3700 1 -- 172.21.15.149:0/2296168109 wait complete. 2024-02-12T16:36:19.677 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:18.337395+0000 mgr.smithi104.ohwbhf (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:19.673567+0000 mon.smithi104 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.149:0/2296168109' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:21.391 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:21.391 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:22.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:20.337686+0000 mgr.smithi104.ohwbhf (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:23.250 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.246+0000 7fe8affff700 1 -- 172.21.15.149:0/4253989148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8b00601e0 msgr2=0x7fe8b00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:23.250 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.246+0000 7fe8affff700 1 --2- 172.21.15.149:0/4253989148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8b00601e0 0x7fe8b00f8f20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fe8a0009fb0 tx=0x7fe8a0009b30).stop 2024-02-12T16:36:23.250 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.246+0000 7fe8affff700 1 -- 172.21.15.149:0/4253989148 shutdown_connections 2024-02-12T16:36:23.250 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.246+0000 7fe8affff700 1 --2- 172.21.15.149:0/4253989148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8b00601e0 0x7fe8b00f8f20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:23.250 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.246+0000 7fe8affff700 1 -- 172.21.15.149:0/4253989148 >> 172.21.15.149:0/4253989148 conn(0x7fe8b00f09a0 msgr2=0x7fe8b00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:23.250 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.246+0000 7fe8affff700 1 -- 172.21.15.149:0/4253989148 shutdown_connections 2024-02-12T16:36:23.250 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.246+0000 7fe8affff700 1 -- 172.21.15.149:0/4253989148 wait complete. 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.247+0000 7fe8affff700 1 Processor -- start 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.247+0000 7fe8affff700 1 -- start start 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.247+0000 7fe8affff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8b00601e0 0x7fe8b00f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.247+0000 7fe8affff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b00f4e20 con 0x7fe8b00601e0 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.247+0000 7fe8aeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8b00601e0 0x7fe8b00f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.247+0000 7fe8aeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8b00601e0 0x7fe8b00f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55456/0 (socket says 172.21.15.149:55456) 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.247+0000 7fe8aeffd700 1 -- 172.21.15.149:0/3223089180 learned_addr learned my addr 172.21.15.149:0/3223089180 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.247+0000 7fe8aeffd700 1 -- 172.21.15.149:0/3223089180 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a0005040 con 0x7fe8b00601e0 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.248+0000 7fe8aeffd700 1 --2- 172.21.15.149:0/3223089180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8b00601e0 0x7fe8b00f69a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fe8a0005010 tx=0x7fe8a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.248+0000 7fe8977fe700 1 -- 172.21.15.149:0/3223089180 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8a0009320 con 0x7fe8b00601e0 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.248+0000 7fe8977fe700 1 -- 172.21.15.149:0/3223089180 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8a0013380 con 0x7fe8b00601e0 2024-02-12T16:36:23.251 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.248+0000 7fe8affff700 1 -- 172.21.15.149:0/3223089180 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b00f4fc0 con 0x7fe8b00601e0 2024-02-12T16:36:23.252 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.248+0000 7fe8977fe700 1 -- 172.21.15.149:0/3223089180 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8a0014e50 con 0x7fe8b00601e0 2024-02-12T16:36:23.252 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.248+0000 7fe8affff700 1 -- 172.21.15.149:0/3223089180 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b00f5450 con 0x7fe8b00601e0 2024-02-12T16:36:23.253 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.249+0000 7fe8977fe700 1 -- 172.21.15.149:0/3223089180 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8a00095f0 con 0x7fe8b00601e0 2024-02-12T16:36:23.253 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.249+0000 7fe8977fe700 1 --2- 172.21.15.149:0/3223089180 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe89802ff10 0x7fe8980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:23.253 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.249+0000 7fe8affff700 1 -- 172.21.15.149:0/3223089180 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b0059100 con 0x7fe8b00601e0 2024-02-12T16:36:23.253 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.249+0000 7fe8977fe700 1 -- 172.21.15.149:0/3223089180 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8a0040e70 con 0x7fe8b00601e0 2024-02-12T16:36:23.253 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.249+0000 7fe8ae7fc700 1 --2- 172.21.15.149:0/3223089180 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe89802ff10 0x7fe8980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:23.253 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.249+0000 7fe8ae7fc700 1 --2- 172.21.15.149:0/3223089180 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe89802ff10 0x7fe8980323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe8a40096a0 tx=0x7fe8a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:23.255 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.253+0000 7fe8977fe700 1 -- 172.21.15.149:0/3223089180 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8a0014420 con 0x7fe8b00601e0 2024-02-12T16:36:23.461 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.459+0000 7fe8affff700 1 -- 172.21.15.149:0/3223089180 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8b00fbe10 con 0x7fe8b00601e0 2024-02-12T16:36:23.462 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.460+0000 7fe8977fe700 1 -- 172.21.15.149:0/3223089180 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe8a0010230 con 0x7fe8b00601e0 2024-02-12T16:36:23.462 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:23.462 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:23.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.462+0000 7fe8affff700 1 -- 172.21.15.149:0/3223089180 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe89802ff10 msgr2=0x7fe8980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:23.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.462+0000 7fe8affff700 1 --2- 172.21.15.149:0/3223089180 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe89802ff10 0x7fe8980323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe8a40096a0 tx=0x7fe8a4006b40).stop 2024-02-12T16:36:23.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.462+0000 7fe8affff700 1 -- 172.21.15.149:0/3223089180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8b00601e0 msgr2=0x7fe8b00f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:23.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.462+0000 7fe8affff700 1 --2- 172.21.15.149:0/3223089180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8b00601e0 0x7fe8b00f69a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fe8a0005010 tx=0x7fe8a000b040).stop 2024-02-12T16:36:23.464 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.462+0000 7fe8affff700 1 -- 172.21.15.149:0/3223089180 shutdown_connections 2024-02-12T16:36:23.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.462+0000 7fe8affff700 1 --2- 172.21.15.149:0/3223089180 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe89802ff10 0x7fe8980323d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:23.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.462+0000 7fe8affff700 1 --2- 172.21.15.149:0/3223089180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8b00601e0 0x7fe8b00f69a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:23.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.462+0000 7fe8affff700 1 -- 172.21.15.149:0/3223089180 >> 172.21.15.149:0/3223089180 conn(0x7fe8b00f09a0 msgr2=0x7fe8b00f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:23.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.462+0000 7fe8affff700 1 -- 172.21.15.149:0/3223089180 shutdown_connections 2024-02-12T16:36:23.465 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:23.462+0000 7fe8affff700 1 -- 172.21.15.149:0/3223089180 wait complete. 2024-02-12T16:36:23.466 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:22.337946+0000 mgr.smithi104.ohwbhf (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:23.461406+0000 mon.smithi104 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.149:0/3223089180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:25.231 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:25.231 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:24.409708+0000 mon.smithi104 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:36:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:24.415399+0000 mon.smithi104 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:36:26.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:24.338331+0000 mgr.smithi104.ohwbhf (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:27.235 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.230+0000 7faf3659a700 1 -- 172.21.15.149:0/1903719611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300f5410 msgr2=0x7faf300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:27.235 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.230+0000 7faf3659a700 1 --2- 172.21.15.149:0/1903719611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300f5410 0x7faf300f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7faf24009fb0 tx=0x7faf24009b30).stop 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.230+0000 7faf3659a700 1 -- 172.21.15.149:0/1903719611 shutdown_connections 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.230+0000 7faf3659a700 1 --2- 172.21.15.149:0/1903719611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300f5410 0x7faf300f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.230+0000 7faf3659a700 1 -- 172.21.15.149:0/1903719611 >> 172.21.15.149:0/1903719611 conn(0x7faf300f0b70 msgr2=0x7faf300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.230+0000 7faf3659a700 1 -- 172.21.15.149:0/1903719611 shutdown_connections 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.230+0000 7faf3659a700 1 -- 172.21.15.149:0/1903719611 wait complete. 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.231+0000 7faf3659a700 1 Processor -- start 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.231+0000 7faf3659a700 1 -- start start 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.231+0000 7faf3659a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300f5410 0x7faf301094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.231+0000 7faf3659a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf30060240 con 0x7faf300f5410 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.231+0000 7faf2ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300f5410 0x7faf301094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.231+0000 7faf2ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300f5410 0x7faf301094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:49318/0 (socket says 172.21.15.149:49318) 2024-02-12T16:36:27.236 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.231+0000 7faf2ffff700 1 -- 172.21.15.149:0/3786520507 learned_addr learned my addr 172.21.15.149:0/3786520507 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:27.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.231+0000 7faf2ffff700 1 -- 172.21.15.149:0/3786520507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf24005040 con 0x7faf300f5410 2024-02-12T16:36:27.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.232+0000 7faf2ffff700 1 --2- 172.21.15.149:0/3786520507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300f5410 0x7faf301094e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7faf240089f0 tx=0x7faf2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:27.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.232+0000 7faf2cff9700 1 -- 172.21.15.149:0/3786520507 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf24009320 con 0x7faf300f5410 2024-02-12T16:36:27.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.232+0000 7faf2cff9700 1 -- 172.21.15.149:0/3786520507 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf24013380 con 0x7faf300f5410 2024-02-12T16:36:27.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.232+0000 7faf2cff9700 1 -- 172.21.15.149:0/3786520507 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf2401ed80 con 0x7faf300f5410 2024-02-12T16:36:27.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.232+0000 7faf3659a700 1 -- 172.21.15.149:0/3786520507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf30060440 con 0x7faf300f5410 2024-02-12T16:36:27.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.232+0000 7faf3659a700 1 -- 172.21.15.149:0/3786520507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf30109b90 con 0x7faf300f5410 2024-02-12T16:36:27.240 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.233+0000 7faf2cff9700 1 -- 172.21.15.149:0/3786520507 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf2401e420 con 0x7faf300f5410 2024-02-12T16:36:27.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.233+0000 7faf3659a700 1 -- 172.21.15.149:0/3786520507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf30045ad0 con 0x7faf300f5410 2024-02-12T16:36:27.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.233+0000 7faf2cff9700 1 --2- 172.21.15.149:0/3786520507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf1802ff60 0x7faf18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:27.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.233+0000 7faf2cff9700 1 -- 172.21.15.149:0/3786520507 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf24041150 con 0x7faf300f5410 2024-02-12T16:36:27.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.233+0000 7faf2f7fe700 1 --2- 172.21.15.149:0/3786520507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf1802ff60 0x7faf18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:27.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.233+0000 7faf2f7fe700 1 --2- 172.21.15.149:0/3786520507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf1802ff60 0x7faf18032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7faf200096a0 tx=0x7faf20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:27.241 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.238+0000 7faf2cff9700 1 -- 172.21.15.149:0/3786520507 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf240095f0 con 0x7faf300f5410 2024-02-12T16:36:27.446 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.444+0000 7faf3659a700 1 -- 172.21.15.149:0/3786520507 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf3010a3b0 con 0x7faf300f5410 2024-02-12T16:36:27.447 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.445+0000 7faf2cff9700 1 -- 172.21.15.149:0/3786520507 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faf24004490 con 0x7faf300f5410 2024-02-12T16:36:27.447 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:27.447 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:27.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.446+0000 7faf3659a700 1 -- 172.21.15.149:0/3786520507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf1802ff60 msgr2=0x7faf18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:27.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.446+0000 7faf3659a700 1 --2- 172.21.15.149:0/3786520507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf1802ff60 0x7faf18032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7faf200096a0 tx=0x7faf20006b40).stop 2024-02-12T16:36:27.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.446+0000 7faf3659a700 1 -- 172.21.15.149:0/3786520507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300f5410 msgr2=0x7faf301094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:27.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.446+0000 7faf3659a700 1 --2- 172.21.15.149:0/3786520507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300f5410 0x7faf301094e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7faf240089f0 tx=0x7faf2400b040).stop 2024-02-12T16:36:27.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.446+0000 7faf3659a700 1 -- 172.21.15.149:0/3786520507 shutdown_connections 2024-02-12T16:36:27.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.446+0000 7faf3659a700 1 --2- 172.21.15.149:0/3786520507 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7faf1802ff60 0x7faf18032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:27.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.446+0000 7faf3659a700 1 --2- 172.21.15.149:0/3786520507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf300f5410 0x7faf301094e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:27.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.446+0000 7faf3659a700 1 -- 172.21.15.149:0/3786520507 >> 172.21.15.149:0/3786520507 conn(0x7faf300f0b70 msgr2=0x7faf300f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:27.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.447+0000 7faf3659a700 1 -- 172.21.15.149:0/3786520507 shutdown_connections 2024-02-12T16:36:27.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:27.447+0000 7faf3659a700 1 -- 172.21.15.149:0/3786520507 wait complete. 2024-02-12T16:36:27.450 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:28.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:26.338679+0000 mgr.smithi104.ohwbhf (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:28.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:27.445994+0000 mon.smithi104 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.149:0/3786520507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:29.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:29.099 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:30.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:28.338941+0000 mgr.smithi104.ohwbhf (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:30.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.933+0000 7f9c3759e700 1 -- 172.21.15.149:0/284961966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c380f62a0 msgr2=0x7f9c380f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:30.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.933+0000 7f9c3759e700 1 --2- 172.21.15.149:0/284961966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c380f62a0 0x7f9c380f66c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f9c2c009fb0 tx=0x7f9c2c009b30).stop 2024-02-12T16:36:30.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.934+0000 7f9c3759e700 1 -- 172.21.15.149:0/284961966 shutdown_connections 2024-02-12T16:36:30.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.934+0000 7f9c3759e700 1 --2- 172.21.15.149:0/284961966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c380f62a0 0x7f9c380f66c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:30.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.934+0000 7f9c3759e700 1 -- 172.21.15.149:0/284961966 >> 172.21.15.149:0/284961966 conn(0x7f9c380f1ac0 msgr2=0x7f9c380f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:30.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.934+0000 7f9c3759e700 1 -- 172.21.15.149:0/284961966 shutdown_connections 2024-02-12T16:36:30.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.934+0000 7f9c3759e700 1 -- 172.21.15.149:0/284961966 wait complete. 2024-02-12T16:36:30.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.934+0000 7f9c3759e700 1 Processor -- start 2024-02-12T16:36:30.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.934+0000 7f9c3759e700 1 -- start start 2024-02-12T16:36:30.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c3759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c380f62a0 0x7f9c38106570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:30.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c3759e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c38106a60 con 0x7f9c380f62a0 2024-02-12T16:36:30.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c3659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c380f62a0 0x7f9c38106570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:30.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c3659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c380f62a0 0x7f9c38106570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:49336/0 (socket says 172.21.15.149:49336) 2024-02-12T16:36:30.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c3659c700 1 -- 172.21.15.149:0/2957417376 learned_addr learned my addr 172.21.15.149:0/2957417376 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:30.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c3659c700 1 -- 172.21.15.149:0/2957417376 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c005040 con 0x7f9c380f62a0 2024-02-12T16:36:30.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c3659c700 1 --2- 172.21.15.149:0/2957417376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c380f62a0 0x7f9c38106570 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f9c2c0089f0 tx=0x7f9c2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:30.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c26ffd700 1 -- 172.21.15.149:0/2957417376 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c2c009320 con 0x7f9c380f62a0 2024-02-12T16:36:30.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c3759e700 1 -- 172.21.15.149:0/2957417376 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c38106c60 con 0x7f9c380f62a0 2024-02-12T16:36:30.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c26ffd700 1 -- 172.21.15.149:0/2957417376 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c2c013380 con 0x7f9c380f62a0 2024-02-12T16:36:30.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c26ffd700 1 -- 172.21.15.149:0/2957417376 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c2c014e20 con 0x7f9c380f62a0 2024-02-12T16:36:30.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.935+0000 7f9c3759e700 1 -- 172.21.15.149:0/2957417376 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c381070f0 con 0x7f9c380f62a0 2024-02-12T16:36:30.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.936+0000 7f9c26ffd700 1 -- 172.21.15.149:0/2957417376 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c2c0263a0 con 0x7f9c380f62a0 2024-02-12T16:36:30.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.936+0000 7f9c26ffd700 1 --2- 172.21.15.149:0/2957417376 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9c2002ff60 0x7f9c20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:30.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.936+0000 7f9c26ffd700 1 -- 172.21.15.149:0/2957417376 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c2c040fb0 con 0x7f9c380f62a0 2024-02-12T16:36:30.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.936+0000 7f9c3759e700 1 -- 172.21.15.149:0/2957417376 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c38045ad0 con 0x7f9c380f62a0 2024-02-12T16:36:30.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.937+0000 7f9c35d9b700 1 --2- 172.21.15.149:0/2957417376 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9c2002ff60 0x7f9c20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:30.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.937+0000 7f9c35d9b700 1 --2- 172.21.15.149:0/2957417376 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9c2002ff60 0x7f9c20032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f9c280096a0 tx=0x7f9c28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:30.944 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:30.941+0000 7f9c26ffd700 1 -- 172.21.15.149:0/2957417376 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c2c012920 con 0x7f9c380f62a0 2024-02-12T16:36:31.146 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.145+0000 7f9c3759e700 1 -- 172.21.15.149:0/2957417376 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c380fbf20 con 0x7f9c380f62a0 2024-02-12T16:36:31.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.145+0000 7f9c26ffd700 1 -- 172.21.15.149:0/2957417376 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9c2c0126f0 con 0x7f9c380f62a0 2024-02-12T16:36:31.148 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:31.148 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:31.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.147+0000 7f9c3759e700 1 -- 172.21.15.149:0/2957417376 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9c2002ff60 msgr2=0x7f9c20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:31.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.147+0000 7f9c3759e700 1 --2- 172.21.15.149:0/2957417376 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9c2002ff60 0x7f9c20032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f9c280096a0 tx=0x7f9c28006b40).stop 2024-02-12T16:36:31.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.147+0000 7f9c3759e700 1 -- 172.21.15.149:0/2957417376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c380f62a0 msgr2=0x7f9c38106570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:31.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.147+0000 7f9c3759e700 1 --2- 172.21.15.149:0/2957417376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c380f62a0 0x7f9c38106570 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f9c2c0089f0 tx=0x7f9c2c00b040).stop 2024-02-12T16:36:31.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.147+0000 7f9c3759e700 1 -- 172.21.15.149:0/2957417376 shutdown_connections 2024-02-12T16:36:31.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.147+0000 7f9c3759e700 1 --2- 172.21.15.149:0/2957417376 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f9c2002ff60 0x7f9c20032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:31.150 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.147+0000 7f9c3759e700 1 --2- 172.21.15.149:0/2957417376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c380f62a0 0x7f9c38106570 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:31.150 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.147+0000 7f9c3759e700 1 -- 172.21.15.149:0/2957417376 >> 172.21.15.149:0/2957417376 conn(0x7f9c380f1ac0 msgr2=0x7f9c380f24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:31.150 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.147+0000 7f9c3759e700 1 -- 172.21.15.149:0/2957417376 shutdown_connections 2024-02-12T16:36:31.150 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:31.147+0000 7f9c3759e700 1 -- 172.21.15.149:0/2957417376 wait complete. 2024-02-12T16:36:31.150 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:31.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:31 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:31.146642+0000 mon.smithi104 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.149:0/2957417376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:32.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:30.339222+0000 mgr.smithi104.ohwbhf (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:32.732 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:32.732 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:34.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:32.339529+0000 mgr.smithi104.ohwbhf (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:34.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.686+0000 7f0bc9757700 1 -- 172.21.15.149:0/691411525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bc40f5410 msgr2=0x7f0bc40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:34.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.686+0000 7f0bc9757700 1 --2- 172.21.15.149:0/691411525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bc40f5410 0x7f0bc40f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f0bac009fb0 tx=0x7f0bac009b30).stop 2024-02-12T16:36:34.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.686+0000 7f0bc9757700 1 -- 172.21.15.149:0/691411525 shutdown_connections 2024-02-12T16:36:34.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.686+0000 7f0bc9757700 1 --2- 172.21.15.149:0/691411525 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bc40f5410 0x7f0bc40f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:34.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.686+0000 7f0bc9757700 1 -- 172.21.15.149:0/691411525 >> 172.21.15.149:0/691411525 conn(0x7f0bc40f0b70 msgr2=0x7f0bc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:34.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.686+0000 7f0bc9757700 1 -- 172.21.15.149:0/691411525 shutdown_connections 2024-02-12T16:36:34.689 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.686+0000 7f0bc9757700 1 -- 172.21.15.149:0/691411525 wait complete. 2024-02-12T16:36:34.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.687+0000 7f0bc9757700 1 Processor -- start 2024-02-12T16:36:34.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.687+0000 7f0bc9757700 1 -- start start 2024-02-12T16:36:34.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.687+0000 7f0bc9757700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bc40f5410 0x7f0bc4105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:34.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.687+0000 7f0bc9757700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc4105c10 con 0x7f0bc40f5410 2024-02-12T16:36:34.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.687+0000 7f0bc2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bc40f5410 0x7f0bc4105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:34.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.687+0000 7f0bc2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bc40f5410 0x7f0bc4105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50798/0 (socket says 172.21.15.149:50798) 2024-02-12T16:36:34.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.687+0000 7f0bc2ffd700 1 -- 172.21.15.149:0/727906038 learned_addr learned my addr 172.21.15.149:0/727906038 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:34.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.688+0000 7f0bc2ffd700 1 -- 172.21.15.149:0/727906038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac005040 con 0x7f0bc40f5410 2024-02-12T16:36:34.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.688+0000 7f0bc2ffd700 1 --2- 172.21.15.149:0/727906038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bc40f5410 0x7f0bc4105720 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f0bac005010 tx=0x7f0bac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:34.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.688+0000 7f0bbb7fe700 1 -- 172.21.15.149:0/727906038 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bac009320 con 0x7f0bc40f5410 2024-02-12T16:36:34.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.688+0000 7f0bc9757700 1 -- 172.21.15.149:0/727906038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc4105e10 con 0x7f0bc40f5410 2024-02-12T16:36:34.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.688+0000 7f0bbb7fe700 1 -- 172.21.15.149:0/727906038 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bac013380 con 0x7f0bc40f5410 2024-02-12T16:36:34.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.688+0000 7f0bbb7fe700 1 -- 172.21.15.149:0/727906038 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bac014e50 con 0x7f0bc40f5410 2024-02-12T16:36:34.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.688+0000 7f0bc9757700 1 -- 172.21.15.149:0/727906038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc41062a0 con 0x7f0bc40f5410 2024-02-12T16:36:34.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.689+0000 7f0bbb7fe700 1 -- 172.21.15.149:0/727906038 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0bac014420 con 0x7f0bc40f5410 2024-02-12T16:36:34.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.689+0000 7f0bc9757700 1 -- 172.21.15.149:0/727906038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc4045ad0 con 0x7f0bc40f5410 2024-02-12T16:36:34.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.689+0000 7f0bbb7fe700 1 --2- 172.21.15.149:0/727906038 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0bb002ff10 0x7f0bb00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:34.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.689+0000 7f0bbb7fe700 1 -- 172.21.15.149:0/727906038 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0bac040e30 con 0x7f0bc40f5410 2024-02-12T16:36:34.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.689+0000 7f0bc27fc700 1 --2- 172.21.15.149:0/727906038 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0bb002ff10 0x7f0bb00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:34.693 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.690+0000 7f0bc27fc700 1 --2- 172.21.15.149:0/727906038 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0bb002ff10 0x7f0bb00323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0bb40096a0 tx=0x7f0bb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:34.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.693+0000 7f0bbb7fe700 1 -- 172.21.15.149:0/727906038 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0bac0095f0 con 0x7f0bc40f5410 2024-02-12T16:36:34.899 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.897+0000 7f0bc9757700 1 -- 172.21.15.149:0/727906038 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0bc4107150 con 0x7f0bc40f5410 2024-02-12T16:36:34.900 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.898+0000 7f0bbb7fe700 1 -- 172.21.15.149:0/727906038 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0bac01db70 con 0x7f0bc40f5410 2024-02-12T16:36:34.900 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:34.900 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:34.902 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.899+0000 7f0bc9757700 1 -- 172.21.15.149:0/727906038 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0bb002ff10 msgr2=0x7f0bb00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:34.902 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.899+0000 7f0bc9757700 1 --2- 172.21.15.149:0/727906038 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0bb002ff10 0x7f0bb00323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0bb40096a0 tx=0x7f0bb4006b40).stop 2024-02-12T16:36:34.902 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.899+0000 7f0bc9757700 1 -- 172.21.15.149:0/727906038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bc40f5410 msgr2=0x7f0bc4105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:34.902 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.899+0000 7f0bc9757700 1 --2- 172.21.15.149:0/727906038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bc40f5410 0x7f0bc4105720 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f0bac005010 tx=0x7f0bac00b040).stop 2024-02-12T16:36:34.902 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.900+0000 7f0bc9757700 1 -- 172.21.15.149:0/727906038 shutdown_connections 2024-02-12T16:36:34.902 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.900+0000 7f0bc9757700 1 --2- 172.21.15.149:0/727906038 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0bb002ff10 0x7f0bb00323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:34.902 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.900+0000 7f0bc9757700 1 --2- 172.21.15.149:0/727906038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0bc40f5410 0x7f0bc4105720 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:34.902 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.900+0000 7f0bc9757700 1 -- 172.21.15.149:0/727906038 >> 172.21.15.149:0/727906038 conn(0x7f0bc40f0b70 msgr2=0x7f0bc40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:34.902 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.900+0000 7f0bc9757700 1 -- 172.21.15.149:0/727906038 shutdown_connections 2024-02-12T16:36:34.902 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:34.900+0000 7f0bc9757700 1 -- 172.21.15.149:0/727906038 wait complete. 2024-02-12T16:36:34.903 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:35.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:35 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:34.899302+0000 mon.smithi104 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.149:0/727906038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:36.557 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:36.557 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:36.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:34.339876+0000 mgr.smithi104.ohwbhf (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:38.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.489+0000 7fde5a8d4700 1 -- 172.21.15.149:0/1190793518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde54060200 msgr2=0x7fde540fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:38.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.489+0000 7fde5a8d4700 1 --2- 172.21.15.149:0/1190793518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde54060200 0x7fde540fb570 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fde44009fb0 tx=0x7fde44009b30).stop 2024-02-12T16:36:38.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.489+0000 7fde5a8d4700 1 -- 172.21.15.149:0/1190793518 shutdown_connections 2024-02-12T16:36:38.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.489+0000 7fde5a8d4700 1 --2- 172.21.15.149:0/1190793518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde54060200 0x7fde540fb570 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:38.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.489+0000 7fde5a8d4700 1 -- 172.21.15.149:0/1190793518 >> 172.21.15.149:0/1190793518 conn(0x7fde540f0b70 msgr2=0x7fde540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:38.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.489+0000 7fde5a8d4700 1 -- 172.21.15.149:0/1190793518 shutdown_connections 2024-02-12T16:36:38.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.489+0000 7fde5a8d4700 1 -- 172.21.15.149:0/1190793518 wait complete. 2024-02-12T16:36:38.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.489+0000 7fde5a8d4700 1 Processor -- start 2024-02-12T16:36:38.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.490+0000 7fde5a8d4700 1 -- start start 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.490+0000 7fde5a8d4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde54060200 0x7fde541089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.490+0000 7fde5a8d4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde54108e90 con 0x7fde54060200 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.490+0000 7fde53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde54060200 0x7fde541089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.490+0000 7fde53fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde54060200 0x7fde541089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50804/0 (socket says 172.21.15.149:50804) 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.490+0000 7fde53fff700 1 -- 172.21.15.149:0/3370112955 learned_addr learned my addr 172.21.15.149:0/3370112955 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.490+0000 7fde53fff700 1 -- 172.21.15.149:0/3370112955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde44005040 con 0x7fde54060200 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.490+0000 7fde53fff700 1 --2- 172.21.15.149:0/3370112955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde54060200 0x7fde541089a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fde440089f0 tx=0x7fde4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.491+0000 7fde50ff9700 1 -- 172.21.15.149:0/3370112955 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde44009320 con 0x7fde54060200 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.491+0000 7fde5a8d4700 1 -- 172.21.15.149:0/3370112955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde54109090 con 0x7fde54060200 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.491+0000 7fde50ff9700 1 -- 172.21.15.149:0/3370112955 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde44013380 con 0x7fde54060200 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.491+0000 7fde50ff9700 1 -- 172.21.15.149:0/3370112955 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde44014e20 con 0x7fde54060200 2024-02-12T16:36:38.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.491+0000 7fde5a8d4700 1 -- 172.21.15.149:0/3370112955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde54109520 con 0x7fde54060200 2024-02-12T16:36:38.495 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.492+0000 7fde50ff9700 1 -- 172.21.15.149:0/3370112955 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde440095f0 con 0x7fde54060200 2024-02-12T16:36:38.495 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.492+0000 7fde5a8d4700 1 -- 172.21.15.149:0/3370112955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde54045ad0 con 0x7fde54060200 2024-02-12T16:36:38.495 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.492+0000 7fde50ff9700 1 --2- 172.21.15.149:0/3370112955 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fde4002ff10 0x7fde400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:38.496 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.492+0000 7fde50ff9700 1 -- 172.21.15.149:0/3370112955 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde44040ef0 con 0x7fde54060200 2024-02-12T16:36:38.496 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.492+0000 7fde537fe700 1 --2- 172.21.15.149:0/3370112955 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fde4002ff10 0x7fde400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:38.496 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.492+0000 7fde537fe700 1 --2- 172.21.15.149:0/3370112955 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fde4002ff10 0x7fde400323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fde3c0096a0 tx=0x7fde3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:38.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.496+0000 7fde50ff9700 1 -- 172.21.15.149:0/3370112955 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde44014420 con 0x7fde54060200 2024-02-12T16:36:38.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:36.340230+0000 mgr.smithi104.ohwbhf (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:38.706 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.704+0000 7fde5a8d4700 1 -- 172.21.15.149:0/3370112955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde5410a3d0 con 0x7fde54060200 2024-02-12T16:36:38.707 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.705+0000 7fde50ff9700 1 -- 172.21.15.149:0/3370112955 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fde44010230 con 0x7fde54060200 2024-02-12T16:36:38.707 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:38.707 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:38.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.706+0000 7fde5a8d4700 1 -- 172.21.15.149:0/3370112955 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fde4002ff10 msgr2=0x7fde400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:38.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.706+0000 7fde5a8d4700 1 --2- 172.21.15.149:0/3370112955 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fde4002ff10 0x7fde400323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fde3c0096a0 tx=0x7fde3c006b40).stop 2024-02-12T16:36:38.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.706+0000 7fde5a8d4700 1 -- 172.21.15.149:0/3370112955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde54060200 msgr2=0x7fde541089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:38.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.706+0000 7fde5a8d4700 1 --2- 172.21.15.149:0/3370112955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde54060200 0x7fde541089a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fde440089f0 tx=0x7fde4400b040).stop 2024-02-12T16:36:38.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.706+0000 7fde5a8d4700 1 -- 172.21.15.149:0/3370112955 shutdown_connections 2024-02-12T16:36:38.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.706+0000 7fde5a8d4700 1 --2- 172.21.15.149:0/3370112955 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fde4002ff10 0x7fde400323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:38.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.706+0000 7fde5a8d4700 1 --2- 172.21.15.149:0/3370112955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fde54060200 0x7fde541089a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:38.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.706+0000 7fde5a8d4700 1 -- 172.21.15.149:0/3370112955 >> 172.21.15.149:0/3370112955 conn(0x7fde540f0b70 msgr2=0x7fde540f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:38.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.706+0000 7fde5a8d4700 1 -- 172.21.15.149:0/3370112955 shutdown_connections 2024-02-12T16:36:38.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:38.706+0000 7fde5a8d4700 1 -- 172.21.15.149:0/3370112955 wait complete. 2024-02-12T16:36:38.709 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:39.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:39 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:38.706123+0000 mon.smithi104 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.149:0/3370112955' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:40.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:40.374 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:40.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:38.340593+0000 mgr.smithi104.ohwbhf (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:42.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.187+0000 7fa7cfdee700 1 -- 172.21.15.149:0/4033810626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c80f6510 msgr2=0x7fa7c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:42.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.187+0000 7fa7cfdee700 1 --2- 172.21.15.149:0/4033810626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c80f6510 0x7fa7c80f6930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fa7b8009fb0 tx=0x7fa7b8009b30).stop 2024-02-12T16:36:42.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.187+0000 7fa7cfdee700 1 -- 172.21.15.149:0/4033810626 shutdown_connections 2024-02-12T16:36:42.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.187+0000 7fa7cfdee700 1 --2- 172.21.15.149:0/4033810626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c80f6510 0x7fa7c80f6930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:42.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.187+0000 7fa7cfdee700 1 -- 172.21.15.149:0/4033810626 >> 172.21.15.149:0/4033810626 conn(0x7fa7c80f1c70 msgr2=0x7fa7c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:42.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.187+0000 7fa7cfdee700 1 -- 172.21.15.149:0/4033810626 shutdown_connections 2024-02-12T16:36:42.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.187+0000 7fa7cfdee700 1 -- 172.21.15.149:0/4033810626 wait complete. 2024-02-12T16:36:42.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.187+0000 7fa7cfdee700 1 Processor -- start 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.187+0000 7fa7cfdee700 1 -- start start 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.187+0000 7fa7cfdee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c80f6510 0x7fa7c8108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.187+0000 7fa7cfdee700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c8108ef0 con 0x7fa7c80f6510 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.188+0000 7fa7cdb8a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c80f6510 0x7fa7c8108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.188+0000 7fa7cdb8a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c80f6510 0x7fa7c8108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50812/0 (socket says 172.21.15.149:50812) 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.188+0000 7fa7cdb8a700 1 -- 172.21.15.149:0/3993398821 learned_addr learned my addr 172.21.15.149:0/3993398821 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.188+0000 7fa7cdb8a700 1 -- 172.21.15.149:0/3993398821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b8005040 con 0x7fa7c80f6510 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.188+0000 7fa7cdb8a700 1 --2- 172.21.15.149:0/3993398821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c80f6510 0x7fa7c8108a00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fa7b8005010 tx=0x7fa7b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.188+0000 7fa7be7fc700 1 -- 172.21.15.149:0/3993398821 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7b8009320 con 0x7fa7c80f6510 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.188+0000 7fa7cfdee700 1 -- 172.21.15.149:0/3993398821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c81090f0 con 0x7fa7c80f6510 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.188+0000 7fa7be7fc700 1 -- 172.21.15.149:0/3993398821 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7b8013380 con 0x7fa7c80f6510 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.188+0000 7fa7be7fc700 1 -- 172.21.15.149:0/3993398821 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7b8014e50 con 0x7fa7c80f6510 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.188+0000 7fa7cfdee700 1 -- 172.21.15.149:0/3993398821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c8109580 con 0x7fa7c80f6510 2024-02-12T16:36:42.195 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.189+0000 7fa7cfdee700 1 -- 172.21.15.149:0/3993398821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c8045ad0 con 0x7fa7c80f6510 2024-02-12T16:36:42.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.189+0000 7fa7be7fc700 1 -- 172.21.15.149:0/3993398821 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7b80095f0 con 0x7fa7c80f6510 2024-02-12T16:36:42.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.189+0000 7fa7be7fc700 1 --2- 172.21.15.149:0/3993398821 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa7b402ff10 0x7fa7b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:42.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.189+0000 7fa7be7fc700 1 -- 172.21.15.149:0/3993398821 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7b8040e90 con 0x7fa7c80f6510 2024-02-12T16:36:42.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.190+0000 7fa7cd389700 1 --2- 172.21.15.149:0/3993398821 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa7b402ff10 0x7fa7b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:42.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.190+0000 7fa7cd389700 1 --2- 172.21.15.149:0/3993398821 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa7b402ff10 0x7fa7b40323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fa7c40096a0 tx=0x7fa7c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:42.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.194+0000 7fa7be7fc700 1 -- 172.21.15.149:0/3993398821 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7b8014420 con 0x7fa7c80f6510 2024-02-12T16:36:42.400 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.398+0000 7fa7cfdee700 1 -- 172.21.15.149:0/3993398821 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7c810a430 con 0x7fa7c80f6510 2024-02-12T16:36:42.401 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.399+0000 7fa7be7fc700 1 -- 172.21.15.149:0/3993398821 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa7b8010230 con 0x7fa7c80f6510 2024-02-12T16:36:42.401 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:42.401 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:42.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.400+0000 7fa7cfdee700 1 -- 172.21.15.149:0/3993398821 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa7b402ff10 msgr2=0x7fa7b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:42.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.400+0000 7fa7cfdee700 1 --2- 172.21.15.149:0/3993398821 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa7b402ff10 0x7fa7b40323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fa7c40096a0 tx=0x7fa7c4006b40).stop 2024-02-12T16:36:42.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.400+0000 7fa7cfdee700 1 -- 172.21.15.149:0/3993398821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c80f6510 msgr2=0x7fa7c8108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:42.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.400+0000 7fa7cfdee700 1 --2- 172.21.15.149:0/3993398821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c80f6510 0x7fa7c8108a00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fa7b8005010 tx=0x7fa7b800b040).stop 2024-02-12T16:36:42.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.401+0000 7fa7cfdee700 1 -- 172.21.15.149:0/3993398821 shutdown_connections 2024-02-12T16:36:42.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.401+0000 7fa7cfdee700 1 --2- 172.21.15.149:0/3993398821 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa7b402ff10 0x7fa7b40323d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:42.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.401+0000 7fa7cfdee700 1 --2- 172.21.15.149:0/3993398821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7c80f6510 0x7fa7c8108a00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:42.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.401+0000 7fa7cfdee700 1 -- 172.21.15.149:0/3993398821 >> 172.21.15.149:0/3993398821 conn(0x7fa7c80f1c70 msgr2=0x7fa7c80f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:42.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.401+0000 7fa7cfdee700 1 -- 172.21.15.149:0/3993398821 shutdown_connections 2024-02-12T16:36:42.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:42.401+0000 7fa7cfdee700 1 -- 172.21.15.149:0/3993398821 wait complete. 2024-02-12T16:36:42.404 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:42.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:40.340960+0000 mgr.smithi104.ohwbhf (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:43.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:43 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:42.399862+0000 mon.smithi104 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.149:0/3993398821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:44.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:44.114 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:44.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:42.341262+0000 mgr.smithi104.ohwbhf (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:46.012 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.997+0000 7f846a075700 1 -- 172.21.15.149:0/1982419917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84640f6510 msgr2=0x7f84640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:46.012 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.997+0000 7f846a075700 1 --2- 172.21.15.149:0/1982419917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84640f6510 0x7f84640f6930 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f8454009fb0 tx=0x7f8454009b30).stop 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.997+0000 7f846a075700 1 -- 172.21.15.149:0/1982419917 shutdown_connections 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.997+0000 7f846a075700 1 --2- 172.21.15.149:0/1982419917 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84640f6510 0x7f84640f6930 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.997+0000 7f846a075700 1 -- 172.21.15.149:0/1982419917 >> 172.21.15.149:0/1982419917 conn(0x7f84640f1c70 msgr2=0x7f84640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.997+0000 7f846a075700 1 -- 172.21.15.149:0/1982419917 shutdown_connections 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.997+0000 7f846a075700 1 -- 172.21.15.149:0/1982419917 wait complete. 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.997+0000 7f846a075700 1 Processor -- start 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.998+0000 7f846a075700 1 -- start start 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.998+0000 7f846a075700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84640f6510 0x7f84641067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.998+0000 7f846a075700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8464106cd0 con 0x7f84640f6510 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.998+0000 7f84637fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84640f6510 0x7f84641067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.998+0000 7f84637fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84640f6510 0x7f84641067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50490/0 (socket says 172.21.15.149:50490) 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.998+0000 7f84637fe700 1 -- 172.21.15.149:0/3338441165 learned_addr learned my addr 172.21.15.149:0/3338441165 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:46.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.998+0000 7f84637fe700 1 -- 172.21.15.149:0/3338441165 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8454005040 con 0x7f84640f6510 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.999+0000 7f84637fe700 1 --2- 172.21.15.149:0/3338441165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84640f6510 0x7f84641067e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f8454005010 tx=0x7f845400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.999+0000 7f844bfff700 1 -- 172.21.15.149:0/3338441165 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8454009320 con 0x7f84640f6510 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.999+0000 7f846a075700 1 -- 172.21.15.149:0/3338441165 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8464106ed0 con 0x7f84640f6510 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.999+0000 7f844bfff700 1 -- 172.21.15.149:0/3338441165 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8454013380 con 0x7f84640f6510 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.999+0000 7f844bfff700 1 -- 172.21.15.149:0/3338441165 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8454014e50 con 0x7f84640f6510 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:45.999+0000 7f846a075700 1 -- 172.21.15.149:0/3338441165 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8464107360 con 0x7f84640f6510 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.000+0000 7f846a075700 1 -- 172.21.15.149:0/3338441165 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8464045ad0 con 0x7f84640f6510 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.000+0000 7f844bfff700 1 -- 172.21.15.149:0/3338441165 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84540095f0 con 0x7f84640f6510 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.000+0000 7f844bfff700 1 --2- 172.21.15.149:0/3338441165 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f844c02ff10 0x7f844c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.000+0000 7f844bfff700 1 -- 172.21.15.149:0/3338441165 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8454040fa0 con 0x7f84640f6510 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.000+0000 7f8462ffd700 1 --2- 172.21.15.149:0/3338441165 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f844c02ff10 0x7f844c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.000+0000 7f8462ffd700 1 --2- 172.21.15.149:0/3338441165 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f844c02ff10 0x7f844c0323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f84580096a0 tx=0x7f8458006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:46.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.004+0000 7f844bfff700 1 -- 172.21.15.149:0/3338441165 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8454014420 con 0x7f84640f6510 2024-02-12T16:36:46.213 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.211+0000 7f846a075700 1 -- 172.21.15.149:0/3338441165 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8464108210 con 0x7f84640f6510 2024-02-12T16:36:46.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.212+0000 7f844bfff700 1 -- 172.21.15.149:0/3338441165 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8454010230 con 0x7f84640f6510 2024-02-12T16:36:46.214 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:46.214 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:46.215 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.213+0000 7f846a075700 1 -- 172.21.15.149:0/3338441165 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f844c02ff10 msgr2=0x7f844c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:46.215 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.213+0000 7f846a075700 1 --2- 172.21.15.149:0/3338441165 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f844c02ff10 0x7f844c0323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f84580096a0 tx=0x7f8458006b40).stop 2024-02-12T16:36:46.215 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.213+0000 7f846a075700 1 -- 172.21.15.149:0/3338441165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84640f6510 msgr2=0x7f84641067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:46.215 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.213+0000 7f846a075700 1 --2- 172.21.15.149:0/3338441165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84640f6510 0x7f84641067e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f8454005010 tx=0x7f845400b040).stop 2024-02-12T16:36:46.215 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.213+0000 7f846a075700 1 -- 172.21.15.149:0/3338441165 shutdown_connections 2024-02-12T16:36:46.216 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.213+0000 7f846a075700 1 --2- 172.21.15.149:0/3338441165 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f844c02ff10 0x7f844c0323d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:46.216 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.213+0000 7f846a075700 1 --2- 172.21.15.149:0/3338441165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84640f6510 0x7f84641067e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:46.216 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.213+0000 7f846a075700 1 -- 172.21.15.149:0/3338441165 >> 172.21.15.149:0/3338441165 conn(0x7f84640f1c70 msgr2=0x7f84640f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:46.216 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.213+0000 7f846a075700 1 -- 172.21.15.149:0/3338441165 shutdown_connections 2024-02-12T16:36:46.216 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:46.213+0000 7f846a075700 1 -- 172.21.15.149:0/3338441165 wait complete. 2024-02-12T16:36:46.216 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:46.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:44.341557+0000 mgr.smithi104.ohwbhf (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:46.587 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:46.212954+0000 mon.smithi104 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.149:0/3338441165' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:47.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:47.782 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:48.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:46.341834+0000 mgr.smithi104.ohwbhf (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:49.690 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.686+0000 7f50d9fa7700 1 -- 172.21.15.149:0/2474844562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50d40f69d0 msgr2=0x7f50d40f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.686+0000 7f50d9fa7700 1 --2- 172.21.15.149:0/2474844562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50d40f69d0 0x7f50d40f6df0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f50c4009fb0 tx=0x7f50c4009b30).stop 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.686+0000 7f50d9fa7700 1 -- 172.21.15.149:0/2474844562 shutdown_connections 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.686+0000 7f50d9fa7700 1 --2- 172.21.15.149:0/2474844562 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50d40f69d0 0x7f50d40f6df0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.686+0000 7f50d9fa7700 1 -- 172.21.15.149:0/2474844562 >> 172.21.15.149:0/2474844562 conn(0x7f50d40f02e0 msgr2=0x7f50d40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.686+0000 7f50d9fa7700 1 -- 172.21.15.149:0/2474844562 shutdown_connections 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.686+0000 7f50d9fa7700 1 -- 172.21.15.149:0/2474844562 wait complete. 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.687+0000 7f50d9fa7700 1 Processor -- start 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.687+0000 7f50d9fa7700 1 -- start start 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.687+0000 7f50d9fa7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50d40f69d0 0x7f50d40fd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.687+0000 7f50d9fa7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d40fb950 con 0x7f50d40f69d0 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.688+0000 7f50d37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50d40f69d0 0x7f50d40fd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.688+0000 7f50d37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50d40f69d0 0x7f50d40fd4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50500/0 (socket says 172.21.15.149:50500) 2024-02-12T16:36:49.691 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.688+0000 7f50d37fe700 1 -- 172.21.15.149:0/159260353 learned_addr learned my addr 172.21.15.149:0/159260353 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:49.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.688+0000 7f50d37fe700 1 -- 172.21.15.149:0/159260353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c4005040 con 0x7f50d40f69d0 2024-02-12T16:36:49.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.688+0000 7f50d37fe700 1 --2- 172.21.15.149:0/159260353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50d40f69d0 0x7f50d40fd4d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f50c4005010 tx=0x7f50c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:49.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.688+0000 7f50cbfff700 1 -- 172.21.15.149:0/159260353 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50c4009320 con 0x7f50d40f69d0 2024-02-12T16:36:49.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.688+0000 7f50d9fa7700 1 -- 172.21.15.149:0/159260353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d40fbaf0 con 0x7f50d40f69d0 2024-02-12T16:36:49.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.688+0000 7f50cbfff700 1 -- 172.21.15.149:0/159260353 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50c4013380 con 0x7f50d40f69d0 2024-02-12T16:36:49.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.688+0000 7f50cbfff700 1 -- 172.21.15.149:0/159260353 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50c4014e50 con 0x7f50d40f69d0 2024-02-12T16:36:49.692 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.688+0000 7f50d9fa7700 1 -- 172.21.15.149:0/159260353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d40fbf80 con 0x7f50d40f69d0 2024-02-12T16:36:49.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.689+0000 7f50cbfff700 1 -- 172.21.15.149:0/159260353 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50c4014420 con 0x7f50d40f69d0 2024-02-12T16:36:49.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.689+0000 7f50cbfff700 1 --2- 172.21.15.149:0/159260353 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f50c002ff10 0x7f50c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:49.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.689+0000 7f50d9fa7700 1 -- 172.21.15.149:0/159260353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d4059100 con 0x7f50d40f69d0 2024-02-12T16:36:49.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.689+0000 7f50cbfff700 1 -- 172.21.15.149:0/159260353 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50c4040f40 con 0x7f50d40f69d0 2024-02-12T16:36:49.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.689+0000 7f50d2ffd700 1 --2- 172.21.15.149:0/159260353 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f50c002ff10 0x7f50c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:49.694 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.690+0000 7f50d2ffd700 1 --2- 172.21.15.149:0/159260353 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f50c002ff10 0x7f50c00323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f50bc0096a0 tx=0x7f50bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:49.696 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.694+0000 7f50cbfff700 1 -- 172.21.15.149:0/159260353 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50c40095f0 con 0x7f50d40f69d0 2024-02-12T16:36:49.901 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.899+0000 7f50d9fa7700 1 -- 172.21.15.149:0/159260353 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50d40faf90 con 0x7f50d40f69d0 2024-02-12T16:36:49.902 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.900+0000 7f50cbfff700 1 -- 172.21.15.149:0/159260353 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f50c401db70 con 0x7f50d40f69d0 2024-02-12T16:36:49.902 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:49.902 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:49.904 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.902+0000 7f50d9fa7700 1 -- 172.21.15.149:0/159260353 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f50c002ff10 msgr2=0x7f50c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:49.904 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.902+0000 7f50d9fa7700 1 --2- 172.21.15.149:0/159260353 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f50c002ff10 0x7f50c00323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f50bc0096a0 tx=0x7f50bc006b40).stop 2024-02-12T16:36:49.904 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.902+0000 7f50d9fa7700 1 -- 172.21.15.149:0/159260353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50d40f69d0 msgr2=0x7f50d40fd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:49.904 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.902+0000 7f50d9fa7700 1 --2- 172.21.15.149:0/159260353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50d40f69d0 0x7f50d40fd4d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f50c4005010 tx=0x7f50c400b040).stop 2024-02-12T16:36:49.904 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.902+0000 7f50d9fa7700 1 -- 172.21.15.149:0/159260353 shutdown_connections 2024-02-12T16:36:49.904 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.902+0000 7f50d9fa7700 1 --2- 172.21.15.149:0/159260353 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f50c002ff10 0x7f50c00323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:49.904 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.902+0000 7f50d9fa7700 1 --2- 172.21.15.149:0/159260353 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50d40f69d0 0x7f50d40fd4d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:49.904 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.902+0000 7f50d9fa7700 1 -- 172.21.15.149:0/159260353 >> 172.21.15.149:0/159260353 conn(0x7f50d40f02e0 msgr2=0x7f50d40f4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:49.904 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.902+0000 7f50d9fa7700 1 -- 172.21.15.149:0/159260353 shutdown_connections 2024-02-12T16:36:49.905 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:49.902+0000 7f50d9fa7700 1 -- 172.21.15.149:0/159260353 wait complete. 2024-02-12T16:36:49.905 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:50.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:48.342067+0000 mgr.smithi104.ohwbhf (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:50.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:49.901401+0000 mon.smithi104 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.149:0/159260353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:51.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:51.516 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:52.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:50.342357+0000 mgr.smithi104.ohwbhf (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:53.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.488+0000 7f56bde72700 1 -- 172.21.15.149:0/1135997981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56b80f6510 msgr2=0x7f56b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:53.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.488+0000 7f56bde72700 1 --2- 172.21.15.149:0/1135997981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56b80f6510 0x7f56b80f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f56a8009fb0 tx=0x7f56a8009b30).stop 2024-02-12T16:36:53.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.488+0000 7f56bde72700 1 -- 172.21.15.149:0/1135997981 shutdown_connections 2024-02-12T16:36:53.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.488+0000 7f56bde72700 1 --2- 172.21.15.149:0/1135997981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56b80f6510 0x7f56b80f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:53.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.488+0000 7f56bde72700 1 -- 172.21.15.149:0/1135997981 >> 172.21.15.149:0/1135997981 conn(0x7f56b80f1c70 msgr2=0x7f56b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:53.491 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.488+0000 7f56bde72700 1 -- 172.21.15.149:0/1135997981 shutdown_connections 2024-02-12T16:36:53.491 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.488+0000 7f56bde72700 1 -- 172.21.15.149:0/1135997981 wait complete. 2024-02-12T16:36:53.491 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.488+0000 7f56bde72700 1 Processor -- start 2024-02-12T16:36:53.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.488+0000 7f56bde72700 1 -- start start 2024-02-12T16:36:53.496 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.488+0000 7f56bde72700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56b80f6510 0x7f56b81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:53.496 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.488+0000 7f56bde72700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b8106cd0 con 0x7f56b80f6510 2024-02-12T16:36:53.496 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.489+0000 7f56b77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56b80f6510 0x7f56b81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:53.496 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.489+0000 7f56b77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56b80f6510 0x7f56b81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50518/0 (socket says 172.21.15.149:50518) 2024-02-12T16:36:53.496 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.489+0000 7f56b77fe700 1 -- 172.21.15.149:0/4107664428 learned_addr learned my addr 172.21.15.149:0/4107664428 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:53.496 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.489+0000 7f56b77fe700 1 -- 172.21.15.149:0/4107664428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a8005040 con 0x7f56b80f6510 2024-02-12T16:36:53.496 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.489+0000 7f56b77fe700 1 --2- 172.21.15.149:0/4107664428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56b80f6510 0x7f56b81067e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f56a80089f0 tx=0x7f56a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:53.496 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.489+0000 7f569ffff700 1 -- 172.21.15.149:0/4107664428 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a8009320 con 0x7f56b80f6510 2024-02-12T16:36:53.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.489+0000 7f56bde72700 1 -- 172.21.15.149:0/4107664428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b8106ed0 con 0x7f56b80f6510 2024-02-12T16:36:53.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.489+0000 7f56bde72700 1 -- 172.21.15.149:0/4107664428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b8107360 con 0x7f56b80f6510 2024-02-12T16:36:53.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.489+0000 7f569ffff700 1 -- 172.21.15.149:0/4107664428 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56a8013380 con 0x7f56b80f6510 2024-02-12T16:36:53.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.489+0000 7f569ffff700 1 -- 172.21.15.149:0/4107664428 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a801ed80 con 0x7f56b80f6510 2024-02-12T16:36:53.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.490+0000 7f569ffff700 1 -- 172.21.15.149:0/4107664428 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56a801e420 con 0x7f56b80f6510 2024-02-12T16:36:53.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.490+0000 7f56bde72700 1 -- 172.21.15.149:0/4107664428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b8045ad0 con 0x7f56b80f6510 2024-02-12T16:36:53.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.491+0000 7f569ffff700 1 --2- 172.21.15.149:0/4107664428 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f56a002ff60 0x7f56a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:53.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.491+0000 7f569ffff700 1 -- 172.21.15.149:0/4107664428 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56a8041000 con 0x7f56b80f6510 2024-02-12T16:36:53.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.491+0000 7f56b6ffd700 1 --2- 172.21.15.149:0/4107664428 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f56a002ff60 0x7f56a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:53.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.491+0000 7f56b6ffd700 1 --2- 172.21.15.149:0/4107664428 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f56a002ff60 0x7f56a0032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f56ac0096a0 tx=0x7f56ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:53.498 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.495+0000 7f569ffff700 1 -- 172.21.15.149:0/4107664428 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56a80095f0 con 0x7f56b80f6510 2024-02-12T16:36:53.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.700+0000 7f56bde72700 1 -- 172.21.15.149:0/4107664428 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56b8108160 con 0x7f56b80f6510 2024-02-12T16:36:53.702 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.701+0000 7f569ffff700 1 -- 172.21.15.149:0/4107664428 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f56a801cb10 con 0x7f56b80f6510 2024-02-12T16:36:53.703 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:53.703 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:53.703 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.703+0000 7f56bde72700 1 -- 172.21.15.149:0/4107664428 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f56a002ff60 msgr2=0x7f56a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:53.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.703+0000 7f56bde72700 1 --2- 172.21.15.149:0/4107664428 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f56a002ff60 0x7f56a0032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f56ac0096a0 tx=0x7f56ac006b40).stop 2024-02-12T16:36:53.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.703+0000 7f56bde72700 1 -- 172.21.15.149:0/4107664428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56b80f6510 msgr2=0x7f56b81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:53.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.703+0000 7f56bde72700 1 --2- 172.21.15.149:0/4107664428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56b80f6510 0x7f56b81067e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f56a80089f0 tx=0x7f56a800b040).stop 2024-02-12T16:36:53.704 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.703+0000 7f56bde72700 1 -- 172.21.15.149:0/4107664428 shutdown_connections 2024-02-12T16:36:53.705 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.703+0000 7f56bde72700 1 --2- 172.21.15.149:0/4107664428 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f56a002ff60 0x7f56a0032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:53.705 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.703+0000 7f56bde72700 1 --2- 172.21.15.149:0/4107664428 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56b80f6510 0x7f56b81067e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:53.705 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.703+0000 7f56bde72700 1 -- 172.21.15.149:0/4107664428 >> 172.21.15.149:0/4107664428 conn(0x7f56b80f1c70 msgr2=0x7f56b80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:53.705 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.703+0000 7f56bde72700 1 -- 172.21.15.149:0/4107664428 shutdown_connections 2024-02-12T16:36:53.705 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:53.703+0000 7f56bde72700 1 -- 172.21.15.149:0/4107664428 wait complete. 2024-02-12T16:36:53.705 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:54.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:52.342581+0000 mgr.smithi104.ohwbhf (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:54.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:53.700961+0000 mon.smithi104 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.149:0/4107664428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:55.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:55.432 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:36:56.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:54.342805+0000 mgr.smithi104.ohwbhf (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:57.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.305+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3347920257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea080f5410 msgr2=0x7fea080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:57.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.305+0000 7fea0eb1d700 1 --2- 172.21.15.149:0/3347920257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea080f5410 0x7fea080f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fe9fc009fb0 tx=0x7fe9fc009b30).stop 2024-02-12T16:36:57.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.305+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3347920257 shutdown_connections 2024-02-12T16:36:57.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.305+0000 7fea0eb1d700 1 --2- 172.21.15.149:0/3347920257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea080f5410 0x7fea080f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:57.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.305+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3347920257 >> 172.21.15.149:0/3347920257 conn(0x7fea080f0b70 msgr2=0x7fea080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:57.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.305+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3347920257 shutdown_connections 2024-02-12T16:36:57.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.305+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3347920257 wait complete. 2024-02-12T16:36:57.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.306+0000 7fea0eb1d700 1 Processor -- start 2024-02-12T16:36:57.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.306+0000 7fea0eb1d700 1 -- start start 2024-02-12T16:36:57.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.306+0000 7fea0eb1d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea080f5410 0x7fea080fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:57.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.306+0000 7fea0eb1d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea080fb960 con 0x7fea080f5410 2024-02-12T16:36:57.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.306+0000 7fea0c8b9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea080f5410 0x7fea080fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:57.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.306+0000 7fea0c8b9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea080f5410 0x7fea080fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55316/0 (socket says 172.21.15.149:55316) 2024-02-12T16:36:57.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.306+0000 7fea0c8b9700 1 -- 172.21.15.149:0/3116733561 learned_addr learned my addr 172.21.15.149:0/3116733561 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:36:57.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.307+0000 7fea0c8b9700 1 -- 172.21.15.149:0/3116733561 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc005040 con 0x7fea080f5410 2024-02-12T16:36:57.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.307+0000 7fea0c8b9700 1 --2- 172.21.15.149:0/3116733561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea080f5410 0x7fea080fd4e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fe9fc005010 tx=0x7fe9fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:57.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.307+0000 7fea057fa700 1 -- 172.21.15.149:0/3116733561 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9fc009320 con 0x7fea080f5410 2024-02-12T16:36:57.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.307+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3116733561 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea080fbb00 con 0x7fea080f5410 2024-02-12T16:36:57.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.307+0000 7fea057fa700 1 -- 172.21.15.149:0/3116733561 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9fc013380 con 0x7fea080f5410 2024-02-12T16:36:57.311 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.307+0000 7fea057fa700 1 -- 172.21.15.149:0/3116733561 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9fc014e50 con 0x7fea080f5410 2024-02-12T16:36:57.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.307+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3116733561 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea080fbf90 con 0x7fea080f5410 2024-02-12T16:36:57.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.308+0000 7fea057fa700 1 -- 172.21.15.149:0/3116733561 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9fc0263a0 con 0x7fea080f5410 2024-02-12T16:36:57.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.308+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3116733561 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea08059100 con 0x7fea080f5410 2024-02-12T16:36:57.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.308+0000 7fea057fa700 1 --2- 172.21.15.149:0/3116733561 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe9f802ff10 0x7fe9f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:36:57.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.308+0000 7fea057fa700 1 -- 172.21.15.149:0/3116733561 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9fc041250 con 0x7fea080f5410 2024-02-12T16:36:57.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.308+0000 7fea07fff700 1 --2- 172.21.15.149:0/3116733561 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe9f802ff10 0x7fe9f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:36:57.312 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.309+0000 7fea07fff700 1 --2- 172.21.15.149:0/3116733561 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe9f802ff10 0x7fe9f80323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe9f40096a0 tx=0x7fe9f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:36:57.315 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.313+0000 7fea057fa700 1 -- 172.21.15.149:0/3116733561 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9fc01cb10 con 0x7fea080f5410 2024-02-12T16:36:57.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.516+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3116733561 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea080fafa0 con 0x7fea080f5410 2024-02-12T16:36:57.519 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.517+0000 7fea057fa700 1 -- 172.21.15.149:0/3116733561 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe9fc01cb10 con 0x7fea080f5410 2024-02-12T16:36:57.519 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:36:57.519 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:36:57.520 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.518+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3116733561 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe9f802ff10 msgr2=0x7fe9f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:57.520 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.518+0000 7fea0eb1d700 1 --2- 172.21.15.149:0/3116733561 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe9f802ff10 0x7fe9f80323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe9f40096a0 tx=0x7fe9f4006b40).stop 2024-02-12T16:36:57.521 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.518+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3116733561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea080f5410 msgr2=0x7fea080fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:36:57.521 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.518+0000 7fea0eb1d700 1 --2- 172.21.15.149:0/3116733561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea080f5410 0x7fea080fd4e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fe9fc005010 tx=0x7fe9fc00b040).stop 2024-02-12T16:36:57.521 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.518+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3116733561 shutdown_connections 2024-02-12T16:36:57.521 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.518+0000 7fea0eb1d700 1 --2- 172.21.15.149:0/3116733561 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe9f802ff10 0x7fe9f80323d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:57.521 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.518+0000 7fea0eb1d700 1 --2- 172.21.15.149:0/3116733561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fea080f5410 0x7fea080fd4e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:36:57.521 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.518+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3116733561 >> 172.21.15.149:0/3116733561 conn(0x7fea080f0b70 msgr2=0x7fea0805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:36:57.521 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.518+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3116733561 shutdown_connections 2024-02-12T16:36:57.521 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:36:57.518+0000 7fea0eb1d700 1 -- 172.21.15.149:0/3116733561 wait complete. 2024-02-12T16:36:57.522 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:36:58.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:56.343098+0000 mgr.smithi104.ohwbhf (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:36:58.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:36:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:36:57.517935+0000 mon.smithi104 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.149:0/3116733561' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:36:59.135 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:36:59.135 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:00.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:36:58.343406+0000 mgr.smithi104.ohwbhf (mgr.14162) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.169+0000 7f064c78a700 1 -- 172.21.15.149:0/4116381343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06440f5410 msgr2=0x7f06440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.169+0000 7f064c78a700 1 --2- 172.21.15.149:0/4116381343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06440f5410 0x7f06440f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f0634009fb0 tx=0x7f0634009b30).stop 2024-02-12T16:37:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.169+0000 7f064c78a700 1 -- 172.21.15.149:0/4116381343 shutdown_connections 2024-02-12T16:37:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.169+0000 7f064c78a700 1 --2- 172.21.15.149:0/4116381343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06440f5410 0x7f06440f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.169+0000 7f064c78a700 1 -- 172.21.15.149:0/4116381343 >> 172.21.15.149:0/4116381343 conn(0x7f06440f0b70 msgr2=0x7f06440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.169+0000 7f064c78a700 1 -- 172.21.15.149:0/4116381343 shutdown_connections 2024-02-12T16:37:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.169+0000 7f064c78a700 1 -- 172.21.15.149:0/4116381343 wait complete. 2024-02-12T16:37:01.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.170+0000 7f064c78a700 1 Processor -- start 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.170+0000 7f064c78a700 1 -- start start 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.170+0000 7f064c78a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06440f5410 0x7f06440fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.170+0000 7f064c78a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06440fb960 con 0x7f06440f5410 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.170+0000 7f064a526700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06440f5410 0x7f06440fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.170+0000 7f064a526700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06440f5410 0x7f06440fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55346/0 (socket says 172.21.15.149:55346) 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.170+0000 7f064a526700 1 -- 172.21.15.149:0/1478161579 learned_addr learned my addr 172.21.15.149:0/1478161579 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.171+0000 7f064a526700 1 -- 172.21.15.149:0/1478161579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0634005040 con 0x7f06440f5410 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.171+0000 7f064a526700 1 --2- 172.21.15.149:0/1478161579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06440f5410 0x7f06440fd4e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f0634005010 tx=0x7f063400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.171+0000 7f063affd700 1 -- 172.21.15.149:0/1478161579 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0634009320 con 0x7f06440f5410 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.171+0000 7f064c78a700 1 -- 172.21.15.149:0/1478161579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06440fbb00 con 0x7f06440f5410 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.171+0000 7f063affd700 1 -- 172.21.15.149:0/1478161579 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0634013380 con 0x7f06440f5410 2024-02-12T16:37:01.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.171+0000 7f063affd700 1 -- 172.21.15.149:0/1478161579 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0634014e50 con 0x7f06440f5410 2024-02-12T16:37:01.175 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.171+0000 7f064c78a700 1 -- 172.21.15.149:0/1478161579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06440fbf90 con 0x7f06440f5410 2024-02-12T16:37:01.175 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.172+0000 7f063affd700 1 -- 172.21.15.149:0/1478161579 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06340263a0 con 0x7f06440f5410 2024-02-12T16:37:01.175 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.172+0000 7f063affd700 1 --2- 172.21.15.149:0/1478161579 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063002ff10 0x7f06300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:01.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.172+0000 7f063affd700 1 -- 172.21.15.149:0/1478161579 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0634040f80 con 0x7f06440f5410 2024-02-12T16:37:01.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.172+0000 7f064c78a700 1 -- 172.21.15.149:0/1478161579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0644059100 con 0x7f06440f5410 2024-02-12T16:37:01.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.172+0000 7f0649d25700 1 --2- 172.21.15.149:0/1478161579 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063002ff10 0x7f06300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:01.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.172+0000 7f0649d25700 1 --2- 172.21.15.149:0/1478161579 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063002ff10 0x7f06300323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f06400096a0 tx=0x7f0640006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:01.178 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.177+0000 7f063affd700 1 -- 172.21.15.149:0/1478161579 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0634012920 con 0x7f06440f5410 2024-02-12T16:37:01.385 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.383+0000 7f064c78a700 1 -- 172.21.15.149:0/1478161579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06440fc690 con 0x7f06440f5410 2024-02-12T16:37:01.385 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.383+0000 7f063affd700 1 -- 172.21.15.149:0/1478161579 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f063402b020 con 0x7f06440f5410 2024-02-12T16:37:01.386 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:01.386 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:01.387 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.385+0000 7f064c78a700 1 -- 172.21.15.149:0/1478161579 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063002ff10 msgr2=0x7f06300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:01.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.385+0000 7f064c78a700 1 --2- 172.21.15.149:0/1478161579 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063002ff10 0x7f06300323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f06400096a0 tx=0x7f0640006b40).stop 2024-02-12T16:37:01.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.385+0000 7f064c78a700 1 -- 172.21.15.149:0/1478161579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06440f5410 msgr2=0x7f06440fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:01.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.385+0000 7f064c78a700 1 --2- 172.21.15.149:0/1478161579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06440f5410 0x7f06440fd4e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f0634005010 tx=0x7f063400b040).stop 2024-02-12T16:37:01.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.385+0000 7f064c78a700 1 -- 172.21.15.149:0/1478161579 shutdown_connections 2024-02-12T16:37:01.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.385+0000 7f064c78a700 1 --2- 172.21.15.149:0/1478161579 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f063002ff10 0x7f06300323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:01.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.385+0000 7f064c78a700 1 --2- 172.21.15.149:0/1478161579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f06440f5410 0x7f06440fd4e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:01.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.385+0000 7f064c78a700 1 -- 172.21.15.149:0/1478161579 >> 172.21.15.149:0/1478161579 conn(0x7f06440f0b70 msgr2=0x7f064405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:01.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.385+0000 7f064c78a700 1 -- 172.21.15.149:0/1478161579 shutdown_connections 2024-02-12T16:37:01.388 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:01.385+0000 7f064c78a700 1 -- 172.21.15.149:0/1478161579 wait complete. 2024-02-12T16:37:01.389 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:02.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:00.343714+0000 mgr.smithi104.ohwbhf (mgr.14162) 234 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:02.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:01.384719+0000 mon.smithi104 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.149:0/1478161579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:03.102 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:03.103 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:04.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:02.344048+0000 mgr.smithi104.ohwbhf (mgr.14162) 235 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:04.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.936+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/2911781103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00f5410 msgr2=0x7fa9c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:04.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.936+0000 7fa9c5b6a700 1 --2- 172.21.15.149:0/2911781103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00f5410 0x7fa9c00f5830 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fa9b4009fb0 tx=0x7fa9b4009b30).stop 2024-02-12T16:37:04.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.936+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/2911781103 shutdown_connections 2024-02-12T16:37:04.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.936+0000 7fa9c5b6a700 1 --2- 172.21.15.149:0/2911781103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00f5410 0x7fa9c00f5830 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:04.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.936+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/2911781103 >> 172.21.15.149:0/2911781103 conn(0x7fa9c00f0b70 msgr2=0x7fa9c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:04.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.936+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/2911781103 shutdown_connections 2024-02-12T16:37:04.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.936+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/2911781103 wait complete. 2024-02-12T16:37:04.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.937+0000 7fa9c5b6a700 1 Processor -- start 2024-02-12T16:37:04.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.937+0000 7fa9c5b6a700 1 -- start start 2024-02-12T16:37:04.940 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.937+0000 7fa9c5b6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00f5410 0x7fa9c0109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:04.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.937+0000 7fa9c5b6a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c0060220 con 0x7fa9c00f5410 2024-02-12T16:37:04.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.937+0000 7fa9bf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00f5410 0x7fa9c0109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:04.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.938+0000 7fa9bf7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00f5410 0x7fa9c0109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:44602/0 (socket says 172.21.15.149:44602) 2024-02-12T16:37:04.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.938+0000 7fa9bf7fe700 1 -- 172.21.15.149:0/1012115368 learned_addr learned my addr 172.21.15.149:0/1012115368 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:04.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.938+0000 7fa9bf7fe700 1 -- 172.21.15.149:0/1012115368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b4005040 con 0x7fa9c00f5410 2024-02-12T16:37:04.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.938+0000 7fa9bf7fe700 1 --2- 172.21.15.149:0/1012115368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00f5410 0x7fa9c0109500 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fa9b4005010 tx=0x7fa9b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:04.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.938+0000 7fa9a7fff700 1 -- 172.21.15.149:0/1012115368 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9b4009320 con 0x7fa9c00f5410 2024-02-12T16:37:04.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.938+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/1012115368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c0060420 con 0x7fa9c00f5410 2024-02-12T16:37:04.942 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.938+0000 7fa9a7fff700 1 -- 172.21.15.149:0/1012115368 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9b4013380 con 0x7fa9c00f5410 2024-02-12T16:37:04.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.938+0000 7fa9a7fff700 1 -- 172.21.15.149:0/1012115368 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9b4014e50 con 0x7fa9c00f5410 2024-02-12T16:37:04.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.938+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/1012115368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c0109bb0 con 0x7fa9c00f5410 2024-02-12T16:37:04.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.939+0000 7fa9a7fff700 1 -- 172.21.15.149:0/1012115368 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9b4014420 con 0x7fa9c00f5410 2024-02-12T16:37:04.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.939+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/1012115368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c0059100 con 0x7fa9c00f5410 2024-02-12T16:37:04.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.939+0000 7fa9a7fff700 1 --2- 172.21.15.149:0/1012115368 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa9a802ff10 0x7fa9a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:04.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.939+0000 7fa9a7fff700 1 -- 172.21.15.149:0/1012115368 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9b4040f40 con 0x7fa9c00f5410 2024-02-12T16:37:04.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.940+0000 7fa9beffd700 1 --2- 172.21.15.149:0/1012115368 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa9a802ff10 0x7fa9a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:04.943 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.940+0000 7fa9beffd700 1 --2- 172.21.15.149:0/1012115368 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa9a802ff10 0x7fa9a80323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fa9b00096a0 tx=0x7fa9b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:04.946 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:04.944+0000 7fa9a7fff700 1 -- 172.21.15.149:0/1012115368 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9b40095f0 con 0x7fa9c00f5410 2024-02-12T16:37:05.149 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.147+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/1012115368 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9c0045ad0 con 0x7fa9c00f5410 2024-02-12T16:37:05.150 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.148+0000 7fa9a7fff700 1 -- 172.21.15.149:0/1012115368 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa9b401db70 con 0x7fa9c00f5410 2024-02-12T16:37:05.150 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:05.150 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:05.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.149+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/1012115368 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa9a802ff10 msgr2=0x7fa9a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:05.152 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.149+0000 7fa9c5b6a700 1 --2- 172.21.15.149:0/1012115368 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa9a802ff10 0x7fa9a80323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fa9b00096a0 tx=0x7fa9b0006b40).stop 2024-02-12T16:37:05.152 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.149+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/1012115368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00f5410 msgr2=0x7fa9c0109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:05.152 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.149+0000 7fa9c5b6a700 1 --2- 172.21.15.149:0/1012115368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00f5410 0x7fa9c0109500 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fa9b4005010 tx=0x7fa9b400b040).stop 2024-02-12T16:37:05.152 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.149+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/1012115368 shutdown_connections 2024-02-12T16:37:05.152 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.149+0000 7fa9c5b6a700 1 --2- 172.21.15.149:0/1012115368 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa9a802ff10 0x7fa9a80323d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:05.152 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.149+0000 7fa9c5b6a700 1 --2- 172.21.15.149:0/1012115368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00f5410 0x7fa9c0109500 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:05.152 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.149+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/1012115368 >> 172.21.15.149:0/1012115368 conn(0x7fa9c00f0b70 msgr2=0x7fa9c00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:05.152 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.150+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/1012115368 shutdown_connections 2024-02-12T16:37:05.152 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:05.150+0000 7fa9c5b6a700 1 -- 172.21.15.149:0/1012115368 wait complete. 2024-02-12T16:37:05.153 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:05.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:05.149318+0000 mon.smithi104 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.149:0/1012115368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:06.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:04.344419+0000 mgr.smithi104.ohwbhf (mgr.14162) 236 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:06.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:06.802 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:08.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.607+0000 7f7125ab5700 1 -- 172.21.15.149:0/3283605267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71200f51a0 msgr2=0x7f71200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:08.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.607+0000 7f7125ab5700 1 --2- 172.21.15.149:0/3283605267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71200f51a0 0x7f71200f55c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f7114009fb0 tx=0x7f7114009b30).stop 2024-02-12T16:37:08.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.607+0000 7f7125ab5700 1 -- 172.21.15.149:0/3283605267 shutdown_connections 2024-02-12T16:37:08.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.607+0000 7f7125ab5700 1 --2- 172.21.15.149:0/3283605267 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71200f51a0 0x7f71200f55c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:08.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.607+0000 7f7125ab5700 1 -- 172.21.15.149:0/3283605267 >> 172.21.15.149:0/3283605267 conn(0x7f71200f09a0 msgr2=0x7f71200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:08.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.607+0000 7f7125ab5700 1 -- 172.21.15.149:0/3283605267 shutdown_connections 2024-02-12T16:37:08.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.607+0000 7f7125ab5700 1 -- 172.21.15.149:0/3283605267 wait complete. 2024-02-12T16:37:08.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.607+0000 7f7125ab5700 1 Processor -- start 2024-02-12T16:37:08.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.607+0000 7f7125ab5700 1 -- start start 2024-02-12T16:37:08.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f7125ab5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71200f51a0 0x7f71201054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:08.614 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f7125ab5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71201059c0 con 0x7f71200f51a0 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f7124ab3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71200f51a0 0x7f71201054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f7124ab3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71200f51a0 0x7f71201054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:44612/0 (socket says 172.21.15.149:44612) 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f7124ab3700 1 -- 172.21.15.149:0/3673102886 learned_addr learned my addr 172.21.15.149:0/3673102886 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f7124ab3700 1 -- 172.21.15.149:0/3673102886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7114005040 con 0x7f71200f51a0 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f7124ab3700 1 --2- 172.21.15.149:0/3673102886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71200f51a0 0x7f71201054d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f71140089f0 tx=0x7f711400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f711d7fa700 1 -- 172.21.15.149:0/3673102886 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7114009320 con 0x7f71200f51a0 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f7125ab5700 1 -- 172.21.15.149:0/3673102886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7120105bc0 con 0x7f71200f51a0 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f711d7fa700 1 -- 172.21.15.149:0/3673102886 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7114013380 con 0x7f71200f51a0 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f711d7fa700 1 -- 172.21.15.149:0/3673102886 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7114014e20 con 0x7f71200f51a0 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.608+0000 7f7125ab5700 1 -- 172.21.15.149:0/3673102886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7120106050 con 0x7f71200f51a0 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.609+0000 7f711d7fa700 1 -- 172.21.15.149:0/3673102886 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f711400f410 con 0x7f71200f51a0 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.609+0000 7f711d7fa700 1 --2- 172.21.15.149:0/3673102886 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f711002ff10 0x7f71100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.609+0000 7f711d7fa700 1 -- 172.21.15.149:0/3673102886 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7114042d20 con 0x7f71200f51a0 2024-02-12T16:37:08.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.609+0000 7f7125ab5700 1 -- 172.21.15.149:0/3673102886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7120045ad0 con 0x7f71200f51a0 2024-02-12T16:37:08.616 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.610+0000 7f711ffff700 1 --2- 172.21.15.149:0/3673102886 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f711002ff10 0x7f71100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:08.616 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.610+0000 7f711ffff700 1 --2- 172.21.15.149:0/3673102886 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f711002ff10 0x7f71100323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f710c0096a0 tx=0x7f710c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:08.616 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.614+0000 7f711d7fa700 1 -- 172.21.15.149:0/3673102886 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71140283a0 con 0x7f71200f51a0 2024-02-12T16:37:08.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:06.344694+0000 mgr.smithi104.ohwbhf (mgr.14162) 237 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:08.819 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.817+0000 7f7125ab5700 1 -- 172.21.15.149:0/3673102886 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7120106da0 con 0x7f71200f51a0 2024-02-12T16:37:08.832 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.817+0000 7f711d7fa700 1 -- 172.21.15.149:0/3673102886 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7114012980 con 0x7f71200f51a0 2024-02-12T16:37:08.832 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:08.833 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:08.833 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.819+0000 7f7125ab5700 1 -- 172.21.15.149:0/3673102886 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f711002ff10 msgr2=0x7f71100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:08.833 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.819+0000 7f7125ab5700 1 --2- 172.21.15.149:0/3673102886 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f711002ff10 0x7f71100323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f710c0096a0 tx=0x7f710c006b40).stop 2024-02-12T16:37:08.833 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.819+0000 7f7125ab5700 1 -- 172.21.15.149:0/3673102886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71200f51a0 msgr2=0x7f71201054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:08.833 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.819+0000 7f7125ab5700 1 --2- 172.21.15.149:0/3673102886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71200f51a0 0x7f71201054d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f71140089f0 tx=0x7f711400b040).stop 2024-02-12T16:37:08.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.819+0000 7f7125ab5700 1 -- 172.21.15.149:0/3673102886 shutdown_connections 2024-02-12T16:37:08.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.819+0000 7f7125ab5700 1 --2- 172.21.15.149:0/3673102886 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f711002ff10 0x7f71100323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:08.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.819+0000 7f7125ab5700 1 --2- 172.21.15.149:0/3673102886 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71200f51a0 0x7f71201054d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:08.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.819+0000 7f7125ab5700 1 -- 172.21.15.149:0/3673102886 >> 172.21.15.149:0/3673102886 conn(0x7f71200f09a0 msgr2=0x7f71200f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:08.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.819+0000 7f7125ab5700 1 -- 172.21.15.149:0/3673102886 shutdown_connections 2024-02-12T16:37:08.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:08.819+0000 7f7125ab5700 1 -- 172.21.15.149:0/3673102886 wait complete. 2024-02-12T16:37:08.834 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:09.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:08.818844+0000 mon.smithi104 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.149:0/3673102886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:10.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:10.514 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:10.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:08.345049+0000 mgr.smithi104.ohwbhf (mgr.14162) 238 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:12.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.426+0000 7f1b8bfff700 1 -- 172.21.15.149:0/2904748952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b8c060120 msgr2=0x7f1b8c0f8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:12.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.426+0000 7f1b8bfff700 1 --2- 172.21.15.149:0/2904748952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b8c060120 0x7f1b8c0f8fe0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f1b7c009fb0 tx=0x7f1b7c009b30).stop 2024-02-12T16:37:12.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.426+0000 7f1b8bfff700 1 -- 172.21.15.149:0/2904748952 shutdown_connections 2024-02-12T16:37:12.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.426+0000 7f1b8bfff700 1 --2- 172.21.15.149:0/2904748952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b8c060120 0x7f1b8c0f8fe0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:12.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.426+0000 7f1b8bfff700 1 -- 172.21.15.149:0/2904748952 >> 172.21.15.149:0/2904748952 conn(0x7f1b8c0f09a0 msgr2=0x7f1b8c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:12.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.426+0000 7f1b8bfff700 1 -- 172.21.15.149:0/2904748952 shutdown_connections 2024-02-12T16:37:12.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.426+0000 7f1b8bfff700 1 -- 172.21.15.149:0/2904748952 wait complete. 2024-02-12T16:37:12.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.427+0000 7f1b8bfff700 1 Processor -- start 2024-02-12T16:37:12.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.427+0000 7f1b8bfff700 1 -- start start 2024-02-12T16:37:12.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.427+0000 7f1b8bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b8c060120 0x7f1b8c0f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:12.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.427+0000 7f1b8bfff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0f4ee0 con 0x7f1b8c060120 2024-02-12T16:37:12.431 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.428+0000 7f1b8affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b8c060120 0x7f1b8c0f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:12.431 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.428+0000 7f1b8affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b8c060120 0x7f1b8c0f6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:44622/0 (socket says 172.21.15.149:44622) 2024-02-12T16:37:12.431 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.428+0000 7f1b8affd700 1 -- 172.21.15.149:0/3183446626 learned_addr learned my addr 172.21.15.149:0/3183446626 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:12.431 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.428+0000 7f1b8affd700 1 -- 172.21.15.149:0/3183446626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c005040 con 0x7f1b8c060120 2024-02-12T16:37:12.431 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.428+0000 7f1b8affd700 1 --2- 172.21.15.149:0/3183446626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b8c060120 0x7f1b8c0f6a60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f1b7c005010 tx=0x7f1b7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:12.432 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.428+0000 7f1b737fe700 1 -- 172.21.15.149:0/3183446626 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b7c009320 con 0x7f1b8c060120 2024-02-12T16:37:12.432 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.428+0000 7f1b8bfff700 1 -- 172.21.15.149:0/3183446626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c0f5080 con 0x7f1b8c060120 2024-02-12T16:37:12.433 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.428+0000 7f1b737fe700 1 -- 172.21.15.149:0/3183446626 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b7c013380 con 0x7f1b8c060120 2024-02-12T16:37:12.433 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.428+0000 7f1b737fe700 1 -- 172.21.15.149:0/3183446626 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b7c014e50 con 0x7f1b8c060120 2024-02-12T16:37:12.433 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.428+0000 7f1b8bfff700 1 -- 172.21.15.149:0/3183446626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c0f5510 con 0x7f1b8c060120 2024-02-12T16:37:12.433 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.429+0000 7f1b737fe700 1 -- 172.21.15.149:0/3183446626 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b7c0095f0 con 0x7f1b8c060120 2024-02-12T16:37:12.433 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.429+0000 7f1b8bfff700 1 -- 172.21.15.149:0/3183446626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c059100 con 0x7f1b8c060120 2024-02-12T16:37:12.433 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.429+0000 7f1b737fe700 1 --2- 172.21.15.149:0/3183446626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b7402ff10 0x7f1b740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:12.433 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.429+0000 7f1b737fe700 1 -- 172.21.15.149:0/3183446626 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b7c040f50 con 0x7f1b8c060120 2024-02-12T16:37:12.433 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.429+0000 7f1b8a7fc700 1 --2- 172.21.15.149:0/3183446626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b7402ff10 0x7f1b740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:12.433 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.430+0000 7f1b8a7fc700 1 --2- 172.21.15.149:0/3183446626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b7402ff10 0x7f1b740323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f1b800096a0 tx=0x7f1b80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:12.436 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.434+0000 7f1b737fe700 1 -- 172.21.15.149:0/3183446626 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b7c014420 con 0x7f1b8c060120 2024-02-12T16:37:12.642 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.640+0000 7f1b8bfff700 1 -- 172.21.15.149:0/3183446626 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b8c045ad0 con 0x7f1b8c060120 2024-02-12T16:37:12.643 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.641+0000 7f1b737fe700 1 -- 172.21.15.149:0/3183446626 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1b7c010230 con 0x7f1b8c060120 2024-02-12T16:37:12.643 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:12.643 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:12.644 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.642+0000 7f1b8bfff700 1 -- 172.21.15.149:0/3183446626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b7402ff10 msgr2=0x7f1b740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:12.644 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.642+0000 7f1b8bfff700 1 --2- 172.21.15.149:0/3183446626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b7402ff10 0x7f1b740323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f1b800096a0 tx=0x7f1b80006b40).stop 2024-02-12T16:37:12.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.642+0000 7f1b8bfff700 1 -- 172.21.15.149:0/3183446626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b8c060120 msgr2=0x7f1b8c0f6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:12.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.642+0000 7f1b8bfff700 1 --2- 172.21.15.149:0/3183446626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b8c060120 0x7f1b8c0f6a60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f1b7c005010 tx=0x7f1b7c00b040).stop 2024-02-12T16:37:12.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.642+0000 7f1b8bfff700 1 -- 172.21.15.149:0/3183446626 shutdown_connections 2024-02-12T16:37:12.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.642+0000 7f1b8bfff700 1 --2- 172.21.15.149:0/3183446626 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1b7402ff10 0x7f1b740323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:12.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.642+0000 7f1b8bfff700 1 --2- 172.21.15.149:0/3183446626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b8c060120 0x7f1b8c0f6a60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:12.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.642+0000 7f1b8bfff700 1 -- 172.21.15.149:0/3183446626 >> 172.21.15.149:0/3183446626 conn(0x7f1b8c0f09a0 msgr2=0x7f1b8c0f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:12.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.642+0000 7f1b8bfff700 1 -- 172.21.15.149:0/3183446626 shutdown_connections 2024-02-12T16:37:12.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:12.642+0000 7f1b8bfff700 1 -- 172.21.15.149:0/3183446626 wait complete. 2024-02-12T16:37:12.645 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:12.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:10.345419+0000 mgr.smithi104.ohwbhf (mgr.14162) 239 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:13.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:12.642203+0000 mon.smithi104 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.149:0/3183446626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:14.410 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:14.410 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:14.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:12.345783+0000 mgr.smithi104.ohwbhf (mgr.14162) 240 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:16.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.044+0000 7f74a2364700 1 -- 172.21.15.149:0/2745561817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f749c060220 msgr2=0x7f749c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:16.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.044+0000 7f74a2364700 1 --2- 172.21.15.149:0/2745561817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f749c060220 0x7f749c0f9150 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f7490009fb0 tx=0x7f7490009b30).stop 2024-02-12T16:37:16.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.044+0000 7f74a2364700 1 -- 172.21.15.149:0/2745561817 shutdown_connections 2024-02-12T16:37:16.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.044+0000 7f74a2364700 1 --2- 172.21.15.149:0/2745561817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f749c060220 0x7f749c0f9150 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:16.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.044+0000 7f74a2364700 1 -- 172.21.15.149:0/2745561817 >> 172.21.15.149:0/2745561817 conn(0x7f749c0f0b70 msgr2=0x7f749c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:16.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.044+0000 7f74a2364700 1 -- 172.21.15.149:0/2745561817 shutdown_connections 2024-02-12T16:37:16.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.044+0000 7f74a2364700 1 -- 172.21.15.149:0/2745561817 wait complete. 2024-02-12T16:37:16.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.044+0000 7f74a2364700 1 Processor -- start 2024-02-12T16:37:16.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.045+0000 7f74a2364700 1 -- start start 2024-02-12T16:37:16.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.045+0000 7f74a2364700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f749c060220 0x7f749c1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:16.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.045+0000 7f74a2364700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c106c90 con 0x7f749c060220 2024-02-12T16:37:16.049 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.045+0000 7f749bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f749c060220 0x7f749c1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:16.049 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.045+0000 7f749bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f749c060220 0x7f749c1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:41136/0 (socket says 172.21.15.149:41136) 2024-02-12T16:37:16.049 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.045+0000 7f749bfff700 1 -- 172.21.15.149:0/14143378 learned_addr learned my addr 172.21.15.149:0/14143378 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:16.049 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.045+0000 7f749bfff700 1 -- 172.21.15.149:0/14143378 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7490005040 con 0x7f749c060220 2024-02-12T16:37:16.050 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.046+0000 7f749bfff700 1 --2- 172.21.15.149:0/14143378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f749c060220 0x7f749c1067a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f74900089f0 tx=0x7f749000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:16.050 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.046+0000 7f7498ff9700 1 -- 172.21.15.149:0/14143378 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7490009320 con 0x7f749c060220 2024-02-12T16:37:16.050 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.046+0000 7f74a2364700 1 -- 172.21.15.149:0/14143378 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c106e90 con 0x7f749c060220 2024-02-12T16:37:16.050 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.046+0000 7f7498ff9700 1 -- 172.21.15.149:0/14143378 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7490013380 con 0x7f749c060220 2024-02-12T16:37:16.050 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.046+0000 7f7498ff9700 1 -- 172.21.15.149:0/14143378 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7490014e20 con 0x7f749c060220 2024-02-12T16:37:16.050 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.046+0000 7f74a2364700 1 -- 172.21.15.149:0/14143378 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c107320 con 0x7f749c060220 2024-02-12T16:37:16.050 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.047+0000 7f7498ff9700 1 -- 172.21.15.149:0/14143378 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74900095f0 con 0x7f749c060220 2024-02-12T16:37:16.050 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.047+0000 7f74a2364700 1 -- 172.21.15.149:0/14143378 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c045ad0 con 0x7f749c060220 2024-02-12T16:37:16.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.047+0000 7f7498ff9700 1 --2- 172.21.15.149:0/14143378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f748402ff10 0x7f74840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:16.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.047+0000 7f7498ff9700 1 -- 172.21.15.149:0/14143378 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7490040f60 con 0x7f749c060220 2024-02-12T16:37:16.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.047+0000 7f749b7fe700 1 --2- 172.21.15.149:0/14143378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f748402ff10 0x7f74840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:16.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.047+0000 7f749b7fe700 1 --2- 172.21.15.149:0/14143378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f748402ff10 0x7f74840323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f748c0096a0 tx=0x7f748c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:16.053 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.051+0000 7f7498ff9700 1 -- 172.21.15.149:0/14143378 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7490014420 con 0x7f749c060220 2024-02-12T16:37:16.257 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.255+0000 7f74a2364700 1 -- 172.21.15.149:0/14143378 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f749c1081d0 con 0x7f749c060220 2024-02-12T16:37:16.257 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.255+0000 7f7498ff9700 1 -- 172.21.15.149:0/14143378 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7490010230 con 0x7f749c060220 2024-02-12T16:37:16.258 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:16.258 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:16.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.257+0000 7f74a2364700 1 -- 172.21.15.149:0/14143378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f748402ff10 msgr2=0x7f74840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:16.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.257+0000 7f74a2364700 1 --2- 172.21.15.149:0/14143378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f748402ff10 0x7f74840323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f748c0096a0 tx=0x7f748c006b40).stop 2024-02-12T16:37:16.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.257+0000 7f74a2364700 1 -- 172.21.15.149:0/14143378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f749c060220 msgr2=0x7f749c1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:16.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.257+0000 7f74a2364700 1 --2- 172.21.15.149:0/14143378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f749c060220 0x7f749c1067a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f74900089f0 tx=0x7f749000b040).stop 2024-02-12T16:37:16.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.257+0000 7f74a2364700 1 -- 172.21.15.149:0/14143378 shutdown_connections 2024-02-12T16:37:16.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.257+0000 7f74a2364700 1 --2- 172.21.15.149:0/14143378 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f748402ff10 0x7f74840323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:16.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.257+0000 7f74a2364700 1 --2- 172.21.15.149:0/14143378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f749c060220 0x7f749c1067a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:16.259 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.257+0000 7f74a2364700 1 -- 172.21.15.149:0/14143378 >> 172.21.15.149:0/14143378 conn(0x7f749c0f0b70 msgr2=0x7f749c0f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:16.260 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.257+0000 7f74a2364700 1 -- 172.21.15.149:0/14143378 shutdown_connections 2024-02-12T16:37:16.260 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:16.257+0000 7f74a2364700 1 -- 172.21.15.149:0/14143378 wait complete. 2024-02-12T16:37:16.260 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:16.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:14.346167+0000 mgr.smithi104.ohwbhf (mgr.14162) 241 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:16.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:16.256779+0000 mon.smithi104 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.149:0/14143378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:17.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:17.953 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:18.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:16.346449+0000 mgr.smithi104.ohwbhf (mgr.14162) 242 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:19.766 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.762+0000 7f2f2fa27700 1 -- 172.21.15.149:0/1736250548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f280f6510 msgr2=0x7f2f280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:19.766 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.762+0000 7f2f2fa27700 1 --2- 172.21.15.149:0/1736250548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f280f6510 0x7f2f280f6930 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f2f18009fb0 tx=0x7f2f18009b30).stop 2024-02-12T16:37:19.766 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.762+0000 7f2f2fa27700 1 -- 172.21.15.149:0/1736250548 shutdown_connections 2024-02-12T16:37:19.766 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.762+0000 7f2f2fa27700 1 --2- 172.21.15.149:0/1736250548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f280f6510 0x7f2f280f6930 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:19.766 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.762+0000 7f2f2fa27700 1 -- 172.21.15.149:0/1736250548 >> 172.21.15.149:0/1736250548 conn(0x7f2f280f1c70 msgr2=0x7f2f280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:19.766 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.762+0000 7f2f2fa27700 1 -- 172.21.15.149:0/1736250548 shutdown_connections 2024-02-12T16:37:19.766 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.762+0000 7f2f2fa27700 1 -- 172.21.15.149:0/1736250548 wait complete. 2024-02-12T16:37:19.766 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.762+0000 7f2f2fa27700 1 Processor -- start 2024-02-12T16:37:19.766 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.762+0000 7f2f2fa27700 1 -- start start 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.763+0000 7f2f2fa27700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f280f6510 0x7f2f281067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.763+0000 7f2f2fa27700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f28106cd0 con 0x7f2f280f6510 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.763+0000 7f2f2d7c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f280f6510 0x7f2f281067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.763+0000 7f2f2d7c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f280f6510 0x7f2f281067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:41144/0 (socket says 172.21.15.149:41144) 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.763+0000 7f2f2d7c3700 1 -- 172.21.15.149:0/4127069196 learned_addr learned my addr 172.21.15.149:0/4127069196 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.763+0000 7f2f2d7c3700 1 -- 172.21.15.149:0/4127069196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f18005040 con 0x7f2f280f6510 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.763+0000 7f2f2d7c3700 1 --2- 172.21.15.149:0/4127069196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f280f6510 0x7f2f281067e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f2f18005010 tx=0x7f2f1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.763+0000 7f2f1dffb700 1 -- 172.21.15.149:0/4127069196 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f18009320 con 0x7f2f280f6510 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.763+0000 7f2f2fa27700 1 -- 172.21.15.149:0/4127069196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f28106ed0 con 0x7f2f280f6510 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.763+0000 7f2f1dffb700 1 -- 172.21.15.149:0/4127069196 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f18013380 con 0x7f2f280f6510 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.764+0000 7f2f1dffb700 1 -- 172.21.15.149:0/4127069196 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f18014e50 con 0x7f2f280f6510 2024-02-12T16:37:19.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.764+0000 7f2f2fa27700 1 -- 172.21.15.149:0/4127069196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f28107360 con 0x7f2f280f6510 2024-02-12T16:37:19.768 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.765+0000 7f2f1dffb700 1 -- 172.21.15.149:0/4127069196 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f180095f0 con 0x7f2f280f6510 2024-02-12T16:37:19.768 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.765+0000 7f2f2fa27700 1 -- 172.21.15.149:0/4127069196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f28045ad0 con 0x7f2f280f6510 2024-02-12T16:37:19.768 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.765+0000 7f2f1dffb700 1 --2- 172.21.15.149:0/4127069196 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2f1402ff10 0x7f2f140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:19.768 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.765+0000 7f2f1dffb700 1 -- 172.21.15.149:0/4127069196 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f18040e90 con 0x7f2f280f6510 2024-02-12T16:37:19.768 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.765+0000 7f2f2cfc2700 1 --2- 172.21.15.149:0/4127069196 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2f1402ff10 0x7f2f140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:19.769 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.766+0000 7f2f2cfc2700 1 --2- 172.21.15.149:0/4127069196 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2f1402ff10 0x7f2f140323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f2f240096a0 tx=0x7f2f24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:19.771 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.770+0000 7f2f1dffb700 1 -- 172.21.15.149:0/4127069196 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f1801d990 con 0x7f2f280f6510 2024-02-12T16:37:19.976 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.974+0000 7f2f2fa27700 1 -- 172.21.15.149:0/4127069196 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f281081b0 con 0x7f2f280f6510 2024-02-12T16:37:19.977 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.975+0000 7f2f1dffb700 1 -- 172.21.15.149:0/4127069196 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2f18010230 con 0x7f2f280f6510 2024-02-12T16:37:19.977 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:19.977 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:19.978 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.976+0000 7f2f2fa27700 1 -- 172.21.15.149:0/4127069196 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2f1402ff10 msgr2=0x7f2f140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:19.978 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.976+0000 7f2f2fa27700 1 --2- 172.21.15.149:0/4127069196 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2f1402ff10 0x7f2f140323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f2f240096a0 tx=0x7f2f24006b40).stop 2024-02-12T16:37:19.979 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.976+0000 7f2f2fa27700 1 -- 172.21.15.149:0/4127069196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f280f6510 msgr2=0x7f2f281067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:19.979 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.976+0000 7f2f2fa27700 1 --2- 172.21.15.149:0/4127069196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f280f6510 0x7f2f281067e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f2f18005010 tx=0x7f2f1800b040).stop 2024-02-12T16:37:19.979 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.976+0000 7f2f2fa27700 1 -- 172.21.15.149:0/4127069196 shutdown_connections 2024-02-12T16:37:19.979 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.976+0000 7f2f2fa27700 1 --2- 172.21.15.149:0/4127069196 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2f1402ff10 0x7f2f140323d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:19.979 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.976+0000 7f2f2fa27700 1 --2- 172.21.15.149:0/4127069196 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f280f6510 0x7f2f281067e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:19.979 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.976+0000 7f2f2fa27700 1 -- 172.21.15.149:0/4127069196 >> 172.21.15.149:0/4127069196 conn(0x7f2f280f1c70 msgr2=0x7f2f280f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:19.979 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.976+0000 7f2f2fa27700 1 -- 172.21.15.149:0/4127069196 shutdown_connections 2024-02-12T16:37:19.979 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:19.976+0000 7f2f2fa27700 1 -- 172.21.15.149:0/4127069196 wait complete. 2024-02-12T16:37:19.979 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:18.346757+0000 mgr.smithi104.ohwbhf (mgr.14162) 243 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:19.975766+0000 mon.smithi104 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.149:0/4127069196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:21.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:21.591 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:22.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:20.347094+0000 mgr.smithi104.ohwbhf (mgr.14162) 244 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:23.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.510+0000 7f53d5674700 1 -- 172.21.15.149:0/4011654146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53d00f5410 msgr2=0x7f53d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:23.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.510+0000 7f53d5674700 1 --2- 172.21.15.149:0/4011654146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53d00f5410 0x7f53d00f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f53c4009fb0 tx=0x7f53c4009b30).stop 2024-02-12T16:37:23.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.510+0000 7f53d5674700 1 -- 172.21.15.149:0/4011654146 shutdown_connections 2024-02-12T16:37:23.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.510+0000 7f53d5674700 1 --2- 172.21.15.149:0/4011654146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53d00f5410 0x7f53d00f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:23.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.510+0000 7f53d5674700 1 -- 172.21.15.149:0/4011654146 >> 172.21.15.149:0/4011654146 conn(0x7f53d00f0b70 msgr2=0x7f53d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:23.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.510+0000 7f53d5674700 1 -- 172.21.15.149:0/4011654146 shutdown_connections 2024-02-12T16:37:23.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.510+0000 7f53d5674700 1 -- 172.21.15.149:0/4011654146 wait complete. 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.511+0000 7f53d5674700 1 Processor -- start 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.511+0000 7f53d5674700 1 -- start start 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.511+0000 7f53d5674700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53d00f5410 0x7f53d01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.511+0000 7f53d5674700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d0106cd0 con 0x7f53d00f5410 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.511+0000 7f53ceffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53d00f5410 0x7f53d01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.511+0000 7f53ceffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53d00f5410 0x7f53d01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:41160/0 (socket says 172.21.15.149:41160) 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.511+0000 7f53ceffd700 1 -- 172.21.15.149:0/2403649894 learned_addr learned my addr 172.21.15.149:0/2403649894 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.511+0000 7f53ceffd700 1 -- 172.21.15.149:0/2403649894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c4005040 con 0x7f53d00f5410 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.512+0000 7f53ceffd700 1 --2- 172.21.15.149:0/2403649894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53d00f5410 0x7f53d01067e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f53c4005010 tx=0x7f53c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.512+0000 7f53b77fe700 1 -- 172.21.15.149:0/2403649894 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53c4009320 con 0x7f53d00f5410 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.512+0000 7f53d5674700 1 -- 172.21.15.149:0/2403649894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d0106ed0 con 0x7f53d00f5410 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.512+0000 7f53b77fe700 1 -- 172.21.15.149:0/2403649894 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53c4013380 con 0x7f53d00f5410 2024-02-12T16:37:23.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.512+0000 7f53b77fe700 1 -- 172.21.15.149:0/2403649894 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53c4014e50 con 0x7f53d00f5410 2024-02-12T16:37:23.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.512+0000 7f53d5674700 1 -- 172.21.15.149:0/2403649894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d0107360 con 0x7f53d00f5410 2024-02-12T16:37:23.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.513+0000 7f53b77fe700 1 -- 172.21.15.149:0/2403649894 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53c40095f0 con 0x7f53d00f5410 2024-02-12T16:37:23.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.513+0000 7f53d5674700 1 -- 172.21.15.149:0/2403649894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d0045ad0 con 0x7f53d00f5410 2024-02-12T16:37:23.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.513+0000 7f53b77fe700 1 --2- 172.21.15.149:0/2403649894 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f53b802ff10 0x7f53b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:23.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.513+0000 7f53b77fe700 1 -- 172.21.15.149:0/2403649894 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53c4040e90 con 0x7f53d00f5410 2024-02-12T16:37:23.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.513+0000 7f53ce7fc700 1 --2- 172.21.15.149:0/2403649894 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f53b802ff10 0x7f53b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:23.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.514+0000 7f53ce7fc700 1 --2- 172.21.15.149:0/2403649894 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f53b802ff10 0x7f53b80323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f53c00096a0 tx=0x7f53c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:23.519 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.518+0000 7f53b77fe700 1 -- 172.21.15.149:0/2403649894 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53c4014420 con 0x7f53d00f5410 2024-02-12T16:37:23.723 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.721+0000 7f53d5674700 1 -- 172.21.15.149:0/2403649894 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53d00fafa0 con 0x7f53d00f5410 2024-02-12T16:37:23.723 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.722+0000 7f53b77fe700 1 -- 172.21.15.149:0/2403649894 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f53c4010230 con 0x7f53d00f5410 2024-02-12T16:37:23.724 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:23.724 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:23.725 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.723+0000 7f53d5674700 1 -- 172.21.15.149:0/2403649894 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f53b802ff10 msgr2=0x7f53b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:23.725 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.723+0000 7f53d5674700 1 --2- 172.21.15.149:0/2403649894 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f53b802ff10 0x7f53b80323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f53c00096a0 tx=0x7f53c0006b40).stop 2024-02-12T16:37:23.725 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.723+0000 7f53d5674700 1 -- 172.21.15.149:0/2403649894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53d00f5410 msgr2=0x7f53d01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:23.725 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.723+0000 7f53d5674700 1 --2- 172.21.15.149:0/2403649894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53d00f5410 0x7f53d01067e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f53c4005010 tx=0x7f53c400b040).stop 2024-02-12T16:37:23.725 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.723+0000 7f53d5674700 1 -- 172.21.15.149:0/2403649894 shutdown_connections 2024-02-12T16:37:23.725 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.723+0000 7f53d5674700 1 --2- 172.21.15.149:0/2403649894 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f53b802ff10 0x7f53b80323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:23.726 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.723+0000 7f53d5674700 1 --2- 172.21.15.149:0/2403649894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53d00f5410 0x7f53d01067e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:23.726 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.723+0000 7f53d5674700 1 -- 172.21.15.149:0/2403649894 >> 172.21.15.149:0/2403649894 conn(0x7f53d00f0b70 msgr2=0x7f53d005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:23.726 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.723+0000 7f53d5674700 1 -- 172.21.15.149:0/2403649894 shutdown_connections 2024-02-12T16:37:23.726 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:23.723+0000 7f53d5674700 1 -- 172.21.15.149:0/2403649894 wait complete. 2024-02-12T16:37:23.726 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:22.347479+0000 mgr.smithi104.ohwbhf (mgr.14162) 245 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:23.722845+0000 mon.smithi104 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.149:0/2403649894' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:25.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:25.439 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:24.410585+0000 mon.smithi104 (mon.0) 260 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:37:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:24.416314+0000 mon.smithi104 (mon.0) 261 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:37:26.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:24.347811+0000 mgr.smithi104.ohwbhf (mgr.14162) 246 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:27.193 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.191+0000 7f67cafb3700 1 -- 172.21.15.149:0/382830622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67c40f6510 msgr2=0x7f67c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:27.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.191+0000 7f67cafb3700 1 --2- 172.21.15.149:0/382830622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67c40f6510 0x7f67c40f6930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f67b8009fb0 tx=0x7f67b8009b30).stop 2024-02-12T16:37:27.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.191+0000 7f67cafb3700 1 -- 172.21.15.149:0/382830622 shutdown_connections 2024-02-12T16:37:27.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.191+0000 7f67cafb3700 1 --2- 172.21.15.149:0/382830622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67c40f6510 0x7f67c40f6930 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:27.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.191+0000 7f67cafb3700 1 -- 172.21.15.149:0/382830622 >> 172.21.15.149:0/382830622 conn(0x7f67c40f1c70 msgr2=0x7f67c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:27.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.191+0000 7f67cafb3700 1 -- 172.21.15.149:0/382830622 shutdown_connections 2024-02-12T16:37:27.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.191+0000 7f67cafb3700 1 -- 172.21.15.149:0/382830622 wait complete. 2024-02-12T16:37:27.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.192+0000 7f67cafb3700 1 Processor -- start 2024-02-12T16:37:27.194 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.192+0000 7f67cafb3700 1 -- start start 2024-02-12T16:37:27.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.192+0000 7f67cafb3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67c40f6510 0x7f67c41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:27.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.192+0000 7f67cafb3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c4106cd0 con 0x7f67c40f6510 2024-02-12T16:37:27.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.192+0000 7f67c8d4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67c40f6510 0x7f67c41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:27.196 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.192+0000 7f67c8d4f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67c40f6510 0x7f67c41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:52260/0 (socket says 172.21.15.149:52260) 2024-02-12T16:37:27.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.192+0000 7f67c8d4f700 1 -- 172.21.15.149:0/1941240326 learned_addr learned my addr 172.21.15.149:0/1941240326 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:27.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.193+0000 7f67c8d4f700 1 -- 172.21.15.149:0/1941240326 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b8005040 con 0x7f67c40f6510 2024-02-12T16:37:27.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.193+0000 7f67c8d4f700 1 --2- 172.21.15.149:0/1941240326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67c40f6510 0x7f67c41067e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f67b8005010 tx=0x7f67b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:27.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.193+0000 7f67c17fa700 1 -- 172.21.15.149:0/1941240326 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67b8009320 con 0x7f67c40f6510 2024-02-12T16:37:27.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.193+0000 7f67cafb3700 1 -- 172.21.15.149:0/1941240326 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c4106ed0 con 0x7f67c40f6510 2024-02-12T16:37:27.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.193+0000 7f67c17fa700 1 -- 172.21.15.149:0/1941240326 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67b8013380 con 0x7f67c40f6510 2024-02-12T16:37:27.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.193+0000 7f67c17fa700 1 -- 172.21.15.149:0/1941240326 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67b8014e50 con 0x7f67c40f6510 2024-02-12T16:37:27.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.193+0000 7f67cafb3700 1 -- 172.21.15.149:0/1941240326 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c4107360 con 0x7f67c40f6510 2024-02-12T16:37:27.197 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.194+0000 7f67c17fa700 1 -- 172.21.15.149:0/1941240326 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67b8014420 con 0x7f67c40f6510 2024-02-12T16:37:27.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.194+0000 7f67cafb3700 1 -- 172.21.15.149:0/1941240326 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c4045ad0 con 0x7f67c40f6510 2024-02-12T16:37:27.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.194+0000 7f67c17fa700 1 --2- 172.21.15.149:0/1941240326 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f67b402ff10 0x7f67b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:27.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.194+0000 7f67c17fa700 1 -- 172.21.15.149:0/1941240326 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67b8040e30 con 0x7f67c40f6510 2024-02-12T16:37:27.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.194+0000 7f67c3fff700 1 --2- 172.21.15.149:0/1941240326 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f67b402ff10 0x7f67b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:27.198 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.195+0000 7f67c3fff700 1 --2- 172.21.15.149:0/1941240326 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f67b402ff10 0x7f67b40323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f67b00096a0 tx=0x7f67b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:27.203 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.199+0000 7f67c17fa700 1 -- 172.21.15.149:0/1941240326 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67b80095f0 con 0x7f67c40f6510 2024-02-12T16:37:27.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.404+0000 7f67cafb3700 1 -- 172.21.15.149:0/1941240326 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67c4108210 con 0x7f67c40f6510 2024-02-12T16:37:27.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.404+0000 7f67c17fa700 1 -- 172.21.15.149:0/1941240326 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f67b801db70 con 0x7f67c40f6510 2024-02-12T16:37:27.407 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:27.407 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:27.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.406+0000 7f67cafb3700 1 -- 172.21.15.149:0/1941240326 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f67b402ff10 msgr2=0x7f67b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:27.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.406+0000 7f67cafb3700 1 --2- 172.21.15.149:0/1941240326 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f67b402ff10 0x7f67b40323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f67b00096a0 tx=0x7f67b0006b40).stop 2024-02-12T16:37:27.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.406+0000 7f67cafb3700 1 -- 172.21.15.149:0/1941240326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67c40f6510 msgr2=0x7f67c41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:27.408 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.406+0000 7f67cafb3700 1 --2- 172.21.15.149:0/1941240326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67c40f6510 0x7f67c41067e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f67b8005010 tx=0x7f67b800b040).stop 2024-02-12T16:37:27.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.406+0000 7f67cafb3700 1 -- 172.21.15.149:0/1941240326 shutdown_connections 2024-02-12T16:37:27.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.406+0000 7f67cafb3700 1 --2- 172.21.15.149:0/1941240326 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f67b402ff10 0x7f67b40323d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:27.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.406+0000 7f67cafb3700 1 --2- 172.21.15.149:0/1941240326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f67c40f6510 0x7f67c41067e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:27.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.406+0000 7f67cafb3700 1 -- 172.21.15.149:0/1941240326 >> 172.21.15.149:0/1941240326 conn(0x7f67c40f1c70 msgr2=0x7f67c40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:27.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.406+0000 7f67cafb3700 1 -- 172.21.15.149:0/1941240326 shutdown_connections 2024-02-12T16:37:27.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:27.406+0000 7f67cafb3700 1 -- 172.21.15.149:0/1941240326 wait complete. 2024-02-12T16:37:27.410 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:28.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:26.348212+0000 mgr.smithi104.ohwbhf (mgr.14162) 247 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:28.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:27.405661+0000 mon.smithi104 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.149:0/1941240326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:29.109 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:29.110 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:30.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:28.348484+0000 mgr.smithi104.ohwbhf (mgr.14162) 248 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:31.025 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.021+0000 7fc5864be700 1 -- 172.21.15.149:0/1316994701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5800f5410 msgr2=0x7fc5800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:31.025 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.021+0000 7fc5864be700 1 --2- 172.21.15.149:0/1316994701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5800f5410 0x7fc5800f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fc570009fb0 tx=0x7fc570009b30).stop 2024-02-12T16:37:31.025 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.021+0000 7fc5864be700 1 -- 172.21.15.149:0/1316994701 shutdown_connections 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.021+0000 7fc5864be700 1 --2- 172.21.15.149:0/1316994701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5800f5410 0x7fc5800f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.021+0000 7fc5864be700 1 -- 172.21.15.149:0/1316994701 >> 172.21.15.149:0/1316994701 conn(0x7fc5800f0b70 msgr2=0x7fc5800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.021+0000 7fc5864be700 1 -- 172.21.15.149:0/1316994701 shutdown_connections 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.021+0000 7fc5864be700 1 -- 172.21.15.149:0/1316994701 wait complete. 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.022+0000 7fc5864be700 1 Processor -- start 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.022+0000 7fc5864be700 1 -- start start 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.022+0000 7fc5864be700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5800f5410 0x7fc5801056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.022+0000 7fc5864be700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc580105bd0 con 0x7fc5800f5410 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.022+0000 7fc57ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5800f5410 0x7fc5801056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.022+0000 7fc57ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5800f5410 0x7fc5801056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:52286/0 (socket says 172.21.15.149:52286) 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.022+0000 7fc57ffff700 1 -- 172.21.15.149:0/1877758915 learned_addr learned my addr 172.21.15.149:0/1877758915 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:31.026 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.023+0000 7fc57ffff700 1 -- 172.21.15.149:0/1877758915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc570005040 con 0x7fc5800f5410 2024-02-12T16:37:31.027 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.023+0000 7fc57ffff700 1 --2- 172.21.15.149:0/1877758915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5800f5410 0x7fc5801056e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fc570005010 tx=0x7fc57000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:31.027 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.023+0000 7fc57d7fa700 1 -- 172.21.15.149:0/1877758915 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc570009320 con 0x7fc5800f5410 2024-02-12T16:37:31.027 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.023+0000 7fc57d7fa700 1 -- 172.21.15.149:0/1877758915 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc570013380 con 0x7fc5800f5410 2024-02-12T16:37:31.027 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.023+0000 7fc5864be700 1 -- 172.21.15.149:0/1877758915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc580105dd0 con 0x7fc5800f5410 2024-02-12T16:37:31.027 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.023+0000 7fc57d7fa700 1 -- 172.21.15.149:0/1877758915 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc570014e50 con 0x7fc5800f5410 2024-02-12T16:37:31.027 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.023+0000 7fc5864be700 1 -- 172.21.15.149:0/1877758915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc580106260 con 0x7fc5800f5410 2024-02-12T16:37:31.028 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.024+0000 7fc57d7fa700 1 -- 172.21.15.149:0/1877758915 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5700263a0 con 0x7fc5800f5410 2024-02-12T16:37:31.028 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.024+0000 7fc57d7fa700 1 --2- 172.21.15.149:0/1877758915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc56c02ff10 0x7fc56c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:31.028 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.024+0000 7fc57d7fa700 1 -- 172.21.15.149:0/1877758915 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc570040ea0 con 0x7fc5800f5410 2024-02-12T16:37:31.028 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.024+0000 7fc5864be700 1 -- 172.21.15.149:0/1877758915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc580045ad0 con 0x7fc5800f5410 2024-02-12T16:37:31.028 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.024+0000 7fc577fff700 1 --2- 172.21.15.149:0/1877758915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc56c02ff10 0x7fc56c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:31.028 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.025+0000 7fc577fff700 1 --2- 172.21.15.149:0/1877758915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc56c02ff10 0x7fc56c0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fc5680096a0 tx=0x7fc568006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:31.030 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.029+0000 7fc57d7fa700 1 -- 172.21.15.149:0/1877758915 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc570012920 con 0x7fc5800f5410 2024-02-12T16:37:31.234 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.232+0000 7fc5864be700 1 -- 172.21.15.149:0/1877758915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc580106f80 con 0x7fc5800f5410 2024-02-12T16:37:31.235 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.233+0000 7fc57d7fa700 1 -- 172.21.15.149:0/1877758915 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc570012330 con 0x7fc5800f5410 2024-02-12T16:37:31.235 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:31.235 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:31.237 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.235+0000 7fc5864be700 1 -- 172.21.15.149:0/1877758915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc56c02ff10 msgr2=0x7fc56c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:31.237 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.235+0000 7fc5864be700 1 --2- 172.21.15.149:0/1877758915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc56c02ff10 0x7fc56c0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fc5680096a0 tx=0x7fc568006b40).stop 2024-02-12T16:37:31.237 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.235+0000 7fc5864be700 1 -- 172.21.15.149:0/1877758915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5800f5410 msgr2=0x7fc5801056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:31.237 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.235+0000 7fc5864be700 1 --2- 172.21.15.149:0/1877758915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5800f5410 0x7fc5801056e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fc570005010 tx=0x7fc57000b040).stop 2024-02-12T16:37:31.237 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.235+0000 7fc5864be700 1 -- 172.21.15.149:0/1877758915 shutdown_connections 2024-02-12T16:37:31.237 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.235+0000 7fc5864be700 1 --2- 172.21.15.149:0/1877758915 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc56c02ff10 0x7fc56c0323d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:31.237 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.235+0000 7fc5864be700 1 --2- 172.21.15.149:0/1877758915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5800f5410 0x7fc5801056e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:31.237 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.235+0000 7fc5864be700 1 -- 172.21.15.149:0/1877758915 >> 172.21.15.149:0/1877758915 conn(0x7fc5800f0b70 msgr2=0x7fc5800f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:31.237 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.235+0000 7fc5864be700 1 -- 172.21.15.149:0/1877758915 shutdown_connections 2024-02-12T16:37:31.238 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:31.235+0000 7fc5864be700 1 -- 172.21.15.149:0/1877758915 wait complete. 2024-02-12T16:37:31.238 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:31.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:31 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:31.234405+0000 mon.smithi104 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.149:0/1877758915' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:32.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:30.348758+0000 mgr.smithi104.ohwbhf (mgr.14162) 249 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:32.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:32.866 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:33.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:37:32.214450+0000 mgr.smithi104.ohwbhf (mgr.14162) 250 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi104 ... 2024-02-12T16:37:33.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Verifying port 9100 ... 2024-02-12T16:37:33.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Non-zero exit code 1 from systemctl start ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@node-exporter.smithi104 2024-02-12T16:37:33.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: systemctl: stderr Job for ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@node-exporter.smithi104.service failed because a timeout was exceeded. 2024-02-12T16:37:33.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: systemctl: stderr See "systemctl status ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@node-exporter.smithi104.service" and "journalctl -xe" for details. 2024-02-12T16:37:33.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Traceback (most recent call last): 2024-02-12T16:37:33.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-12T16:37:33.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: main() 2024-02-12T16:37:33.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-12T16:37:33.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: r = ctx.func(ctx) 2024-02-12T16:37:33.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: return func(ctx) 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ports=daemon_ports) 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: RuntimeError: Failed command: systemctl start ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@node-exporter.smithi104 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Traceback (most recent call last): 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-12T16:37:33.677 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: yield (conn, connr) 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: code, '\n'.join(err))) 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi104 ... 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Verifying port 9100 ... 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Non-zero exit code 1 from systemctl start ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@node-exporter.smithi104 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: systemctl: stderr Job for ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@node-exporter.smithi104.service failed because a timeout was exceeded. 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: systemctl: stderr See "systemctl status ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@node-exporter.smithi104.service" and "journalctl -xe" for details. 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Traceback (most recent call last): 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: main() 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: r = ctx.func(ctx) 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-12T16:37:33.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: return func(ctx) 2024-02-12T16:37:33.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-12T16:37:33.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: ports=daemon_ports) 2024-02-12T16:37:33.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-12T16:37:33.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-12T16:37:33.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-12T16:37:33.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-12T16:37:33.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-12T16:37:33.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-12T16:37:33.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: RuntimeError: Failed command: systemctl start ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@node-exporter.smithi104 2024-02-12T16:37:33.679 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:37:32.221640+0000 mgr.smithi104.ohwbhf (mgr.14162) 251 : cephadm [INF] Deploying daemon prometheus.smithi104 on smithi104 2024-02-12T16:37:34.657 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.655+0000 7fddb1238700 1 -- 172.21.15.149:0/669716390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddac0f5410 msgr2=0x7fddac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:34.658 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.655+0000 7fddb1238700 1 --2- 172.21.15.149:0/669716390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddac0f5410 0x7fddac0f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fdda0009fb0 tx=0x7fdda0009b30).stop 2024-02-12T16:37:34.658 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.655+0000 7fddb1238700 1 -- 172.21.15.149:0/669716390 shutdown_connections 2024-02-12T16:37:34.658 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.655+0000 7fddb1238700 1 --2- 172.21.15.149:0/669716390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddac0f5410 0x7fddac0f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:34.658 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.655+0000 7fddb1238700 1 -- 172.21.15.149:0/669716390 >> 172.21.15.149:0/669716390 conn(0x7fddac0f0b70 msgr2=0x7fddac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:34.658 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.655+0000 7fddb1238700 1 -- 172.21.15.149:0/669716390 shutdown_connections 2024-02-12T16:37:34.658 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.655+0000 7fddb1238700 1 -- 172.21.15.149:0/669716390 wait complete. 2024-02-12T16:37:34.660 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.656+0000 7fddb1238700 1 Processor -- start 2024-02-12T16:37:34.660 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.656+0000 7fddb1238700 1 -- start start 2024-02-12T16:37:34.660 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.656+0000 7fddb1238700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddac0f5410 0x7fddac109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:34.660 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.656+0000 7fddb1238700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddac060240 con 0x7fddac0f5410 2024-02-12T16:37:34.660 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.657+0000 7fddaad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddac0f5410 0x7fddac109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.657+0000 7fddaad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddac0f5410 0x7fddac109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33416/0 (socket says 172.21.15.149:33416) 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.657+0000 7fddaad9d700 1 -- 172.21.15.149:0/1814186961 learned_addr learned my addr 172.21.15.149:0/1814186961 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.657+0000 7fddaad9d700 1 -- 172.21.15.149:0/1814186961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda0005040 con 0x7fddac0f5410 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.657+0000 7fddaad9d700 1 --2- 172.21.15.149:0/1814186961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddac0f5410 0x7fddac109520 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fdda0005010 tx=0x7fdda000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.657+0000 7fdd937fe700 1 -- 172.21.15.149:0/1814186961 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdda0009320 con 0x7fddac0f5410 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.657+0000 7fdd937fe700 1 -- 172.21.15.149:0/1814186961 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdda0013380 con 0x7fddac0f5410 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.657+0000 7fddb1238700 1 -- 172.21.15.149:0/1814186961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddac060440 con 0x7fddac0f5410 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.657+0000 7fdd937fe700 1 -- 172.21.15.149:0/1814186961 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdda0014e50 con 0x7fddac0f5410 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.657+0000 7fddb1238700 1 -- 172.21.15.149:0/1814186961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddac109bd0 con 0x7fddac0f5410 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.658+0000 7fdd937fe700 1 -- 172.21.15.149:0/1814186961 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdda00095f0 con 0x7fddac0f5410 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.658+0000 7fddb1238700 1 -- 172.21.15.149:0/1814186961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddac045ad0 con 0x7fddac0f5410 2024-02-12T16:37:34.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.658+0000 7fdd937fe700 1 --2- 172.21.15.149:0/1814186961 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdd9402ff10 0x7fdd940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:34.662 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.658+0000 7fdd937fe700 1 -- 172.21.15.149:0/1814186961 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdda0040fa0 con 0x7fddac0f5410 2024-02-12T16:37:34.662 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.658+0000 7fddaa59c700 1 --2- 172.21.15.149:0/1814186961 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdd9402ff10 0x7fdd940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:34.662 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.659+0000 7fddaa59c700 1 --2- 172.21.15.149:0/1814186961 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdd9402ff10 0x7fdd940323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fdd9c0096a0 tx=0x7fdd9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:34.665 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.663+0000 7fdd937fe700 1 -- 172.21.15.149:0/1814186961 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdda0014420 con 0x7fddac0f5410 2024-02-12T16:37:34.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:32.348997+0000 mgr.smithi104.ohwbhf (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:34.868 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.866+0000 7fddb1238700 1 -- 172.21.15.149:0/1814186961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fddac10a3f0 con 0x7fddac0f5410 2024-02-12T16:37:34.868 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.866+0000 7fdd937fe700 1 -- 172.21.15.149:0/1814186961 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdda0010230 con 0x7fddac0f5410 2024-02-12T16:37:34.869 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:34.869 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:34.870 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.868+0000 7fddb1238700 1 -- 172.21.15.149:0/1814186961 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdd9402ff10 msgr2=0x7fdd940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:34.870 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.868+0000 7fddb1238700 1 --2- 172.21.15.149:0/1814186961 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdd9402ff10 0x7fdd940323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fdd9c0096a0 tx=0x7fdd9c006b40).stop 2024-02-12T16:37:34.870 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.868+0000 7fddb1238700 1 -- 172.21.15.149:0/1814186961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddac0f5410 msgr2=0x7fddac109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:34.871 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.868+0000 7fddb1238700 1 --2- 172.21.15.149:0/1814186961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddac0f5410 0x7fddac109520 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fdda0005010 tx=0x7fdda000b040).stop 2024-02-12T16:37:34.871 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.868+0000 7fddb1238700 1 -- 172.21.15.149:0/1814186961 shutdown_connections 2024-02-12T16:37:34.871 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.868+0000 7fddb1238700 1 --2- 172.21.15.149:0/1814186961 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fdd9402ff10 0x7fdd940323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:34.871 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.868+0000 7fddb1238700 1 --2- 172.21.15.149:0/1814186961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fddac0f5410 0x7fddac109520 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:34.871 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.868+0000 7fddb1238700 1 -- 172.21.15.149:0/1814186961 >> 172.21.15.149:0/1814186961 conn(0x7fddac0f0b70 msgr2=0x7fddac0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:34.871 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.868+0000 7fddb1238700 1 -- 172.21.15.149:0/1814186961 shutdown_connections 2024-02-12T16:37:34.871 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:34.868+0000 7fddb1238700 1 -- 172.21.15.149:0/1814186961 wait complete. 2024-02-12T16:37:34.871 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:35.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:35 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:34.867741+0000 mon.smithi104 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.149:0/1814186961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:36.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:36.633 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:36.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:34.349195+0000 mgr.smithi104.ohwbhf (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:38.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.283+0000 7f623b6bc700 1 -- 172.21.15.149:0/1598371524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62340601e0 msgr2=0x7f62340f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:38.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.283+0000 7f623b6bc700 1 --2- 172.21.15.149:0/1598371524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62340601e0 0x7f62340f9190 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f6224009fb0 tx=0x7f6224009b30).stop 2024-02-12T16:37:38.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.283+0000 7f623b6bc700 1 -- 172.21.15.149:0/1598371524 shutdown_connections 2024-02-12T16:37:38.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.283+0000 7f623b6bc700 1 --2- 172.21.15.149:0/1598371524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62340601e0 0x7f62340f9190 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:38.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.283+0000 7f623b6bc700 1 -- 172.21.15.149:0/1598371524 >> 172.21.15.149:0/1598371524 conn(0x7f62340f0b70 msgr2=0x7f62340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:38.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.283+0000 7f623b6bc700 1 -- 172.21.15.149:0/1598371524 shutdown_connections 2024-02-12T16:37:38.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.283+0000 7f623b6bc700 1 -- 172.21.15.149:0/1598371524 wait complete. 2024-02-12T16:37:38.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.284+0000 7f623b6bc700 1 Processor -- start 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.284+0000 7f623b6bc700 1 -- start start 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.284+0000 7f623b6bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62340601e0 0x7f62340f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.284+0000 7f623b6bc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62340f5090 con 0x7f62340601e0 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.284+0000 7f6239458700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62340601e0 0x7f62340f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.284+0000 7f6239458700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62340601e0 0x7f62340f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33430/0 (socket says 172.21.15.149:33430) 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.284+0000 7f6239458700 1 -- 172.21.15.149:0/1339559426 learned_addr learned my addr 172.21.15.149:0/1339559426 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.284+0000 7f6239458700 1 -- 172.21.15.149:0/1339559426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6224005040 con 0x7f62340601e0 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.285+0000 7f6239458700 1 --2- 172.21.15.149:0/1339559426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62340601e0 0x7f62340f6c10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f6224007a10 tx=0x7f6224009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.285+0000 7f6229ffb700 1 -- 172.21.15.149:0/1339559426 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6224018070 con 0x7f62340601e0 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.285+0000 7f623b6bc700 1 -- 172.21.15.149:0/1339559426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62340f5230 con 0x7f62340601e0 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.285+0000 7f6229ffb700 1 -- 172.21.15.149:0/1339559426 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6224013410 con 0x7f62340601e0 2024-02-12T16:37:38.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.285+0000 7f6229ffb700 1 -- 172.21.15.149:0/1339559426 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f622401c3c0 con 0x7f62340601e0 2024-02-12T16:37:38.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.285+0000 7f623b6bc700 1 -- 172.21.15.149:0/1339559426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62340f56c0 con 0x7f62340601e0 2024-02-12T16:37:38.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.286+0000 7f6229ffb700 1 -- 172.21.15.149:0/1339559426 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f622401c680 con 0x7f62340601e0 2024-02-12T16:37:38.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.286+0000 7f6229ffb700 1 --2- 172.21.15.149:0/1339559426 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f622002ff60 0x7f6220032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:38.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.286+0000 7f6229ffb700 1 -- 172.21.15.149:0/1339559426 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6224041500 con 0x7f62340601e0 2024-02-12T16:37:38.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.286+0000 7f623b6bc700 1 -- 172.21.15.149:0/1339559426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6234059100 con 0x7f62340601e0 2024-02-12T16:37:38.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.286+0000 7f6238c57700 1 --2- 172.21.15.149:0/1339559426 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f622002ff60 0x7f6220032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:38.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.287+0000 7f6238c57700 1 --2- 172.21.15.149:0/1339559426 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f622002ff60 0x7f6220032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f62300096a0 tx=0x7f6230006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:38.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.292+0000 7f6229ffb700 1 -- 172.21.15.149:0/1339559426 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f622401ca40 con 0x7f62340601e0 2024-02-12T16:37:38.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:36.349444+0000 mgr.smithi104.ohwbhf (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:38.499 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.497+0000 7f623b6bc700 1 -- 172.21.15.149:0/1339559426 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6234045ad0 con 0x7f62340601e0 2024-02-12T16:37:38.500 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.498+0000 7f6229ffb700 1 -- 172.21.15.149:0/1339559426 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f622401b9c0 con 0x7f62340601e0 2024-02-12T16:37:38.500 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:38.500 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:38.502 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.499+0000 7f623b6bc700 1 -- 172.21.15.149:0/1339559426 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f622002ff60 msgr2=0x7f6220032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:38.502 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.499+0000 7f623b6bc700 1 --2- 172.21.15.149:0/1339559426 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f622002ff60 0x7f6220032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f62300096a0 tx=0x7f6230006b40).stop 2024-02-12T16:37:38.502 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.499+0000 7f623b6bc700 1 -- 172.21.15.149:0/1339559426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62340601e0 msgr2=0x7f62340f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:38.502 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.499+0000 7f623b6bc700 1 --2- 172.21.15.149:0/1339559426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62340601e0 0x7f62340f6c10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f6224007a10 tx=0x7f6224009b30).stop 2024-02-12T16:37:38.502 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.499+0000 7f623b6bc700 1 -- 172.21.15.149:0/1339559426 shutdown_connections 2024-02-12T16:37:38.502 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.499+0000 7f623b6bc700 1 --2- 172.21.15.149:0/1339559426 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f622002ff60 0x7f6220032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:38.502 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.499+0000 7f623b6bc700 1 --2- 172.21.15.149:0/1339559426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62340601e0 0x7f62340f6c10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:38.502 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.499+0000 7f623b6bc700 1 -- 172.21.15.149:0/1339559426 >> 172.21.15.149:0/1339559426 conn(0x7f62340f0b70 msgr2=0x7f62340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:38.502 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.499+0000 7f623b6bc700 1 -- 172.21.15.149:0/1339559426 shutdown_connections 2024-02-12T16:37:38.502 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:38.499+0000 7f623b6bc700 1 -- 172.21.15.149:0/1339559426 wait complete. 2024-02-12T16:37:38.503 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:39.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:39 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:38.499161+0000 mon.smithi104 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.149:0/1339559426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:40.165 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:40.165 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:40.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:38.349737+0000 mgr.smithi104.ohwbhf (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:42.044 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.041+0000 7f2625b36700 1 -- 172.21.15.149:0/1007429609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2620060220 msgr2=0x7f26200f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:42.044 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.041+0000 7f2625b36700 1 --2- 172.21.15.149:0/1007429609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2620060220 0x7f26200f9150 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f2610009fb0 tx=0x7f2610009b30).stop 2024-02-12T16:37:42.044 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.041+0000 7f2625b36700 1 -- 172.21.15.149:0/1007429609 shutdown_connections 2024-02-12T16:37:42.044 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.042+0000 7f2625b36700 1 --2- 172.21.15.149:0/1007429609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2620060220 0x7f26200f9150 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:42.044 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.042+0000 7f2625b36700 1 -- 172.21.15.149:0/1007429609 >> 172.21.15.149:0/1007429609 conn(0x7f26200f0b70 msgr2=0x7f26200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:42.044 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.042+0000 7f2625b36700 1 -- 172.21.15.149:0/1007429609 shutdown_connections 2024-02-12T16:37:42.044 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.042+0000 7f2625b36700 1 -- 172.21.15.149:0/1007429609 wait complete. 2024-02-12T16:37:42.044 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.042+0000 7f2625b36700 1 Processor -- start 2024-02-12T16:37:42.044 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.042+0000 7f2625b36700 1 -- start start 2024-02-12T16:37:42.045 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.042+0000 7f2625b36700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2620060220 0x7f26201067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:42.045 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.042+0000 7f2625b36700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2620106cb0 con 0x7f2620060220 2024-02-12T16:37:42.045 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.043+0000 7f261f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2620060220 0x7f26201067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:42.045 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.043+0000 7f261f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2620060220 0x7f26201067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:33460/0 (socket says 172.21.15.149:33460) 2024-02-12T16:37:42.045 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.043+0000 7f261f7fe700 1 -- 172.21.15.149:0/77840108 learned_addr learned my addr 172.21.15.149:0/77840108 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:42.045 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.043+0000 7f261f7fe700 1 -- 172.21.15.149:0/77840108 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2610005040 con 0x7f2620060220 2024-02-12T16:37:42.045 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.043+0000 7f261f7fe700 1 --2- 172.21.15.149:0/77840108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2620060220 0x7f26201067c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f26100089f0 tx=0x7f261000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:42.046 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.043+0000 7f2617fff700 1 -- 172.21.15.149:0/77840108 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2610009320 con 0x7f2620060220 2024-02-12T16:37:42.046 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.043+0000 7f2625b36700 1 -- 172.21.15.149:0/77840108 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2620106eb0 con 0x7f2620060220 2024-02-12T16:37:42.046 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.043+0000 7f2617fff700 1 -- 172.21.15.149:0/77840108 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2610013380 con 0x7f2620060220 2024-02-12T16:37:42.046 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.043+0000 7f2617fff700 1 -- 172.21.15.149:0/77840108 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2610014e20 con 0x7f2620060220 2024-02-12T16:37:42.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.043+0000 7f2625b36700 1 -- 172.21.15.149:0/77840108 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2620107340 con 0x7f2620060220 2024-02-12T16:37:42.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.044+0000 7f2617fff700 1 -- 172.21.15.149:0/77840108 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26100263a0 con 0x7f2620060220 2024-02-12T16:37:42.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.044+0000 7f2617fff700 1 --2- 172.21.15.149:0/77840108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f260c02ff60 0x7f260c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:42.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.044+0000 7f2625b36700 1 -- 172.21.15.149:0/77840108 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2620045ad0 con 0x7f2620060220 2024-02-12T16:37:42.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.044+0000 7f2617fff700 1 -- 172.21.15.149:0/77840108 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2610040fa0 con 0x7f2620060220 2024-02-12T16:37:42.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.044+0000 7f261effd700 1 --2- 172.21.15.149:0/77840108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f260c02ff60 0x7f260c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:42.047 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.045+0000 7f261effd700 1 --2- 172.21.15.149:0/77840108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f260c02ff60 0x7f260c032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f26080096a0 tx=0x7f2608006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:42.050 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.049+0000 7f2617fff700 1 -- 172.21.15.149:0/77840108 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2610012920 con 0x7f2620060220 2024-02-12T16:37:42.255 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.254+0000 7f2625b36700 1 -- 172.21.15.149:0/77840108 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2620108190 con 0x7f2620060220 2024-02-12T16:37:42.255 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.254+0000 7f2617fff700 1 -- 172.21.15.149:0/77840108 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f26100126f0 con 0x7f2620060220 2024-02-12T16:37:42.256 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:42.256 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:42.257 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.256+0000 7f2625b36700 1 -- 172.21.15.149:0/77840108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f260c02ff60 msgr2=0x7f260c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:42.258 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.256+0000 7f2625b36700 1 --2- 172.21.15.149:0/77840108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f260c02ff60 0x7f260c032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f26080096a0 tx=0x7f2608006b40).stop 2024-02-12T16:37:42.258 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.256+0000 7f2625b36700 1 -- 172.21.15.149:0/77840108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2620060220 msgr2=0x7f26201067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:42.258 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.256+0000 7f2625b36700 1 --2- 172.21.15.149:0/77840108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2620060220 0x7f26201067c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f26100089f0 tx=0x7f261000b040).stop 2024-02-12T16:37:42.258 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.256+0000 7f2625b36700 1 -- 172.21.15.149:0/77840108 shutdown_connections 2024-02-12T16:37:42.258 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.256+0000 7f2625b36700 1 --2- 172.21.15.149:0/77840108 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f260c02ff60 0x7f260c032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:42.258 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.256+0000 7f2625b36700 1 --2- 172.21.15.149:0/77840108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2620060220 0x7f26201067c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:42.258 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.256+0000 7f2625b36700 1 -- 172.21.15.149:0/77840108 >> 172.21.15.149:0/77840108 conn(0x7f26200f0b70 msgr2=0x7f26200f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:42.258 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.256+0000 7f2625b36700 1 -- 172.21.15.149:0/77840108 shutdown_connections 2024-02-12T16:37:42.258 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:42.256+0000 7f2625b36700 1 -- 172.21.15.149:0/77840108 wait complete. 2024-02-12T16:37:42.258 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:42.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:40.350020+0000 mgr.smithi104.ohwbhf (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:43.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:43 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:42.254620+0000 mon.smithi104 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.149:0/77840108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:44.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:44.029 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:44.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:42.350298+0000 mgr.smithi104.ohwbhf (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:45.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.811+0000 7f5512946700 1 -- 172.21.15.149:0/1047128713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f550c0f5410 msgr2=0x7f550c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:45.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.811+0000 7f5512946700 1 --2- 172.21.15.149:0/1047128713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f550c0f5410 0x7f550c0f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f54f4009fb0 tx=0x7f54f4009b30).stop 2024-02-12T16:37:45.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.811+0000 7f5512946700 1 -- 172.21.15.149:0/1047128713 shutdown_connections 2024-02-12T16:37:45.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.811+0000 7f5512946700 1 --2- 172.21.15.149:0/1047128713 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f550c0f5410 0x7f550c0f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:45.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.811+0000 7f5512946700 1 -- 172.21.15.149:0/1047128713 >> 172.21.15.149:0/1047128713 conn(0x7f550c0f0b70 msgr2=0x7f550c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:45.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.811+0000 7f5512946700 1 -- 172.21.15.149:0/1047128713 shutdown_connections 2024-02-12T16:37:45.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.811+0000 7f5512946700 1 -- 172.21.15.149:0/1047128713 wait complete. 2024-02-12T16:37:45.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.812+0000 7f5512946700 1 Processor -- start 2024-02-12T16:37:45.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.812+0000 7f5512946700 1 -- start start 2024-02-12T16:37:45.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.812+0000 7f5512946700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f550c0f5410 0x7f550c109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:45.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.812+0000 7f5512946700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c060220 con 0x7f550c0f5410 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.812+0000 7f550bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f550c0f5410 0x7f550c109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.812+0000 7f550bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f550c0f5410 0x7f550c109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:45000/0 (socket says 172.21.15.149:45000) 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.812+0000 7f550bfff700 1 -- 172.21.15.149:0/2120771034 learned_addr learned my addr 172.21.15.149:0/2120771034 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.813+0000 7f550bfff700 1 -- 172.21.15.149:0/2120771034 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f4005040 con 0x7f550c0f5410 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.813+0000 7f550bfff700 1 --2- 172.21.15.149:0/2120771034 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f550c0f5410 0x7f550c109530 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f54f4005010 tx=0x7f54f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.813+0000 7f5508ff9700 1 -- 172.21.15.149:0/2120771034 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f4009320 con 0x7f550c0f5410 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.813+0000 7f5508ff9700 1 -- 172.21.15.149:0/2120771034 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54f4013380 con 0x7f550c0f5410 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.813+0000 7f5512946700 1 -- 172.21.15.149:0/2120771034 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550c060420 con 0x7f550c0f5410 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.813+0000 7f5508ff9700 1 -- 172.21.15.149:0/2120771034 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f4014e50 con 0x7f550c0f5410 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.813+0000 7f5512946700 1 -- 172.21.15.149:0/2120771034 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550c109be0 con 0x7f550c0f5410 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.814+0000 7f5508ff9700 1 -- 172.21.15.149:0/2120771034 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54f4014420 con 0x7f550c0f5410 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.814+0000 7f5512946700 1 -- 172.21.15.149:0/2120771034 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550c059100 con 0x7f550c0f5410 2024-02-12T16:37:45.816 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.814+0000 7f5508ff9700 1 --2- 172.21.15.149:0/2120771034 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f54f802ff10 0x7f54f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:45.817 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.814+0000 7f5508ff9700 1 -- 172.21.15.149:0/2120771034 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54f4040e30 con 0x7f550c0f5410 2024-02-12T16:37:45.817 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.814+0000 7f550b7fe700 1 --2- 172.21.15.149:0/2120771034 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f54f802ff10 0x7f54f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:45.817 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.815+0000 7f550b7fe700 1 --2- 172.21.15.149:0/2120771034 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f54f802ff10 0x7f54f80323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f54fc0096a0 tx=0x7f54fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:45.820 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:45.819+0000 7f5508ff9700 1 -- 172.21.15.149:0/2120771034 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54f40095f0 con 0x7f550c0f5410 2024-02-12T16:37:46.022 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.021+0000 7f5512946700 1 -- 172.21.15.149:0/2120771034 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f550c045ad0 con 0x7f550c0f5410 2024-02-12T16:37:46.022 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.021+0000 7f5508ff9700 1 -- 172.21.15.149:0/2120771034 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f54f401db70 con 0x7f550c0f5410 2024-02-12T16:37:46.023 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:46.023 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:46.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.023+0000 7f5512946700 1 -- 172.21.15.149:0/2120771034 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f54f802ff10 msgr2=0x7f54f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:46.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.023+0000 7f5512946700 1 --2- 172.21.15.149:0/2120771034 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f54f802ff10 0x7f54f80323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f54fc0096a0 tx=0x7f54fc006b40).stop 2024-02-12T16:37:46.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.023+0000 7f5512946700 1 -- 172.21.15.149:0/2120771034 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f550c0f5410 msgr2=0x7f550c109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:46.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.023+0000 7f5512946700 1 --2- 172.21.15.149:0/2120771034 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f550c0f5410 0x7f550c109530 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f54f4005010 tx=0x7f54f400b040).stop 2024-02-12T16:37:46.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.023+0000 7f5512946700 1 -- 172.21.15.149:0/2120771034 shutdown_connections 2024-02-12T16:37:46.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.023+0000 7f5512946700 1 --2- 172.21.15.149:0/2120771034 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f54f802ff10 0x7f54f80323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:46.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.023+0000 7f5512946700 1 --2- 172.21.15.149:0/2120771034 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f550c0f5410 0x7f550c109530 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:46.024 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.023+0000 7f5512946700 1 -- 172.21.15.149:0/2120771034 >> 172.21.15.149:0/2120771034 conn(0x7f550c0f0b70 msgr2=0x7f550c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:46.025 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.023+0000 7f5512946700 1 -- 172.21.15.149:0/2120771034 shutdown_connections 2024-02-12T16:37:46.025 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:46.023+0000 7f5512946700 1 -- 172.21.15.149:0/2120771034 wait complete. 2024-02-12T16:37:46.025 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:46.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:44.350689+0000 mgr.smithi104.ohwbhf (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:46.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:46.021623+0000 mon.smithi104 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.149:0/2120771034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:47.721 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:47.722 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:48.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:46.351152+0000 mgr.smithi104.ohwbhf (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:49.619 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.617+0000 7f22d286b700 1 -- 172.21.15.149:0/2283754513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22cc060280 msgr2=0x7f22cc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:49.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.617+0000 7f22d286b700 1 --2- 172.21.15.149:0/2283754513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22cc060280 0x7f22cc0f90f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f22c0009fb0 tx=0x7f22c0009b30).stop 2024-02-12T16:37:49.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.617+0000 7f22d286b700 1 -- 172.21.15.149:0/2283754513 shutdown_connections 2024-02-12T16:37:49.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.617+0000 7f22d286b700 1 --2- 172.21.15.149:0/2283754513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22cc060280 0x7f22cc0f90f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:49.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.617+0000 7f22d286b700 1 -- 172.21.15.149:0/2283754513 >> 172.21.15.149:0/2283754513 conn(0x7f22cc0f0b70 msgr2=0x7f22cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:49.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.617+0000 7f22d286b700 1 -- 172.21.15.149:0/2283754513 shutdown_connections 2024-02-12T16:37:49.620 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.617+0000 7f22d286b700 1 -- 172.21.15.149:0/2283754513 wait complete. 2024-02-12T16:37:49.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.618+0000 7f22d286b700 1 Processor -- start 2024-02-12T16:37:49.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.618+0000 7f22d286b700 1 -- start start 2024-02-12T16:37:49.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.618+0000 7f22d286b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22cc060280 0x7f22cc106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:49.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.618+0000 7f22d286b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc106c70 con 0x7f22cc060280 2024-02-12T16:37:49.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.618+0000 7f22cbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22cc060280 0x7f22cc106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:49.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.618+0000 7f22cbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22cc060280 0x7f22cc106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:45016/0 (socket says 172.21.15.149:45016) 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.618+0000 7f22cbfff700 1 -- 172.21.15.149:0/772756054 learned_addr learned my addr 172.21.15.149:0/772756054 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.619+0000 7f22cbfff700 1 -- 172.21.15.149:0/772756054 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c0005040 con 0x7f22cc060280 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.619+0000 7f22cbfff700 1 --2- 172.21.15.149:0/772756054 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22cc060280 0x7f22cc106780 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f22c00089f0 tx=0x7f22c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.619+0000 7f22c8ff9700 1 -- 172.21.15.149:0/772756054 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22c0009320 con 0x7f22cc060280 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.619+0000 7f22d286b700 1 -- 172.21.15.149:0/772756054 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22cc106e70 con 0x7f22cc060280 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.619+0000 7f22c8ff9700 1 -- 172.21.15.149:0/772756054 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22c0013380 con 0x7f22cc060280 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.619+0000 7f22c8ff9700 1 -- 172.21.15.149:0/772756054 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22c0014e20 con 0x7f22cc060280 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.619+0000 7f22d286b700 1 -- 172.21.15.149:0/772756054 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22cc107300 con 0x7f22cc060280 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.620+0000 7f22c8ff9700 1 -- 172.21.15.149:0/772756054 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22c00263a0 con 0x7f22cc060280 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.620+0000 7f22c8ff9700 1 --2- 172.21.15.149:0/772756054 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f22b402ff10 0x7f22b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.620+0000 7f22c8ff9700 1 -- 172.21.15.149:0/772756054 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f22c0040f70 con 0x7f22cc060280 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.620+0000 7f22d286b700 1 -- 172.21.15.149:0/772756054 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22cc045ad0 con 0x7f22cc060280 2024-02-12T16:37:49.623 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.620+0000 7f22cb7fe700 1 --2- 172.21.15.149:0/772756054 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f22b402ff10 0x7f22b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:49.624 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.621+0000 7f22cb7fe700 1 --2- 172.21.15.149:0/772756054 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f22b402ff10 0x7f22b40323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f22bc0096a0 tx=0x7f22bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:49.627 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.625+0000 7f22c8ff9700 1 -- 172.21.15.149:0/772756054 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22c0012920 con 0x7f22cc060280 2024-02-12T16:37:49.831 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.829+0000 7f22d286b700 1 -- 172.21.15.149:0/772756054 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22cc0fc090 con 0x7f22cc060280 2024-02-12T16:37:49.832 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.830+0000 7f22c8ff9700 1 -- 172.21.15.149:0/772756054 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f22c002b020 con 0x7f22cc060280 2024-02-12T16:37:49.832 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:49.832 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:49.833 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.831+0000 7f22d286b700 1 -- 172.21.15.149:0/772756054 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f22b402ff10 msgr2=0x7f22b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:49.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.831+0000 7f22d286b700 1 --2- 172.21.15.149:0/772756054 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f22b402ff10 0x7f22b40323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f22bc0096a0 tx=0x7f22bc006b40).stop 2024-02-12T16:37:49.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.831+0000 7f22d286b700 1 -- 172.21.15.149:0/772756054 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22cc060280 msgr2=0x7f22cc106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:49.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.831+0000 7f22d286b700 1 --2- 172.21.15.149:0/772756054 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22cc060280 0x7f22cc106780 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f22c00089f0 tx=0x7f22c000b040).stop 2024-02-12T16:37:49.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.831+0000 7f22d286b700 1 -- 172.21.15.149:0/772756054 shutdown_connections 2024-02-12T16:37:49.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.831+0000 7f22d286b700 1 --2- 172.21.15.149:0/772756054 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f22b402ff10 0x7f22b40323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:49.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.831+0000 7f22d286b700 1 --2- 172.21.15.149:0/772756054 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f22cc060280 0x7f22cc106780 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:49.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.831+0000 7f22d286b700 1 -- 172.21.15.149:0/772756054 >> 172.21.15.149:0/772756054 conn(0x7f22cc0f0b70 msgr2=0x7f22cc0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:49.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.831+0000 7f22d286b700 1 -- 172.21.15.149:0/772756054 shutdown_connections 2024-02-12T16:37:49.834 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:49.831+0000 7f22d286b700 1 -- 172.21.15.149:0/772756054 wait complete. 2024-02-12T16:37:49.835 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:50.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:48.351615+0000 mgr.smithi104.ohwbhf (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:50.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:49.831157+0000 mon.smithi104 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.149:0/772756054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:51.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:51.510 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:52.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:50.352008+0000 mgr.smithi104.ohwbhf (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:53.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.211+0000 7f666cf44700 1 -- 172.21.15.149:0/1091336209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66680f5410 msgr2=0x7f66680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:53.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.211+0000 7f666cf44700 1 --2- 172.21.15.149:0/1091336209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66680f5410 0x7f66680f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f665c009fb0 tx=0x7f665c009b30).stop 2024-02-12T16:37:53.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.211+0000 7f666cf44700 1 -- 172.21.15.149:0/1091336209 shutdown_connections 2024-02-12T16:37:53.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.211+0000 7f666cf44700 1 --2- 172.21.15.149:0/1091336209 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66680f5410 0x7f66680f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:53.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.211+0000 7f666cf44700 1 -- 172.21.15.149:0/1091336209 >> 172.21.15.149:0/1091336209 conn(0x7f66680f0b70 msgr2=0x7f66680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:53.214 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.211+0000 7f666cf44700 1 -- 172.21.15.149:0/1091336209 shutdown_connections 2024-02-12T16:37:53.215 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.212+0000 7f666cf44700 1 -- 172.21.15.149:0/1091336209 wait complete. 2024-02-12T16:37:53.215 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.212+0000 7f666cf44700 1 Processor -- start 2024-02-12T16:37:53.215 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.212+0000 7f666cf44700 1 -- start start 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.212+0000 7f666cf44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66680f5410 0x7f66681056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.212+0000 7f666cf44700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6668105bd0 con 0x7f66680f5410 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.213+0000 7f666659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66680f5410 0x7f66681056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.213+0000 7f666659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66680f5410 0x7f66681056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:45032/0 (socket says 172.21.15.149:45032) 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.213+0000 7f666659c700 1 -- 172.21.15.149:0/2820377198 learned_addr learned my addr 172.21.15.149:0/2820377198 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.213+0000 7f666659c700 1 -- 172.21.15.149:0/2820377198 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c005040 con 0x7f66680f5410 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.213+0000 7f666659c700 1 --2- 172.21.15.149:0/2820377198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66680f5410 0x7f66681056e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f665c005010 tx=0x7f665c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.213+0000 7f6656ffd700 1 -- 172.21.15.149:0/2820377198 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f665c009320 con 0x7f66680f5410 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.213+0000 7f666cf44700 1 -- 172.21.15.149:0/2820377198 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6668105dd0 con 0x7f66680f5410 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.213+0000 7f6656ffd700 1 -- 172.21.15.149:0/2820377198 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f665c013380 con 0x7f66680f5410 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.213+0000 7f6656ffd700 1 -- 172.21.15.149:0/2820377198 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f665c014e50 con 0x7f66680f5410 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.213+0000 7f666cf44700 1 -- 172.21.15.149:0/2820377198 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6668106260 con 0x7f66680f5410 2024-02-12T16:37:53.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.214+0000 7f6656ffd700 1 -- 172.21.15.149:0/2820377198 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f665c0095f0 con 0x7f66680f5410 2024-02-12T16:37:53.218 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.214+0000 7f666cf44700 1 -- 172.21.15.149:0/2820377198 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6668045ad0 con 0x7f66680f5410 2024-02-12T16:37:53.218 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.214+0000 7f6656ffd700 1 --2- 172.21.15.149:0/2820377198 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f665002ff10 0x7f66500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:53.218 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.214+0000 7f6656ffd700 1 -- 172.21.15.149:0/2820377198 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f665c042d80 con 0x7f66680f5410 2024-02-12T16:37:53.218 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.215+0000 7f6665d9b700 1 --2- 172.21.15.149:0/2820377198 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f665002ff10 0x7f66500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:53.218 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.215+0000 7f6665d9b700 1 --2- 172.21.15.149:0/2820377198 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f665002ff10 0x7f66500323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f66580096a0 tx=0x7f6658006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:53.221 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.219+0000 7f6656ffd700 1 -- 172.21.15.149:0/2820377198 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f665c014420 con 0x7f66680f5410 2024-02-12T16:37:53.426 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.424+0000 7f666cf44700 1 -- 172.21.15.149:0/2820377198 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6668107110 con 0x7f66680f5410 2024-02-12T16:37:53.427 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.425+0000 7f6656ffd700 1 -- 172.21.15.149:0/2820377198 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f665c011290 con 0x7f66680f5410 2024-02-12T16:37:53.427 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:53.427 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:53.428 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.426+0000 7f666cf44700 1 -- 172.21.15.149:0/2820377198 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f665002ff10 msgr2=0x7f66500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:53.428 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.426+0000 7f666cf44700 1 --2- 172.21.15.149:0/2820377198 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f665002ff10 0x7f66500323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f66580096a0 tx=0x7f6658006b40).stop 2024-02-12T16:37:53.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.426+0000 7f666cf44700 1 -- 172.21.15.149:0/2820377198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66680f5410 msgr2=0x7f66681056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:53.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.426+0000 7f666cf44700 1 --2- 172.21.15.149:0/2820377198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66680f5410 0x7f66681056e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f665c005010 tx=0x7f665c00b040).stop 2024-02-12T16:37:53.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.426+0000 7f666cf44700 1 -- 172.21.15.149:0/2820377198 shutdown_connections 2024-02-12T16:37:53.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.426+0000 7f666cf44700 1 --2- 172.21.15.149:0/2820377198 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f665002ff10 0x7f66500323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:53.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.426+0000 7f666cf44700 1 --2- 172.21.15.149:0/2820377198 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f66680f5410 0x7f66681056e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:53.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.426+0000 7f666cf44700 1 -- 172.21.15.149:0/2820377198 >> 172.21.15.149:0/2820377198 conn(0x7f66680f0b70 msgr2=0x7f66680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:53.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.426+0000 7f666cf44700 1 -- 172.21.15.149:0/2820377198 shutdown_connections 2024-02-12T16:37:53.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:53.426+0000 7f666cf44700 1 -- 172.21.15.149:0/2820377198 wait complete. 2024-02-12T16:37:53.430 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:54.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:52.352328+0000 mgr.smithi104.ohwbhf (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:54.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:53.426140+0000 mon.smithi104 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.149:0/2820377198' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:55.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:55.118 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:37:56.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:54.352744+0000 mgr.smithi104.ohwbhf (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:56.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.932+0000 7f00f5a57700 1 -- 172.21.15.149:0/2763223831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f00f6510 msgr2=0x7f00f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:56.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.932+0000 7f00f5a57700 1 --2- 172.21.15.149:0/2763223831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f00f6510 0x7f00f00f6930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f00e4009fb0 tx=0x7f00e4009b30).stop 2024-02-12T16:37:56.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.932+0000 7f00f5a57700 1 -- 172.21.15.149:0/2763223831 shutdown_connections 2024-02-12T16:37:56.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.932+0000 7f00f5a57700 1 --2- 172.21.15.149:0/2763223831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f00f6510 0x7f00f00f6930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:56.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.932+0000 7f00f5a57700 1 -- 172.21.15.149:0/2763223831 >> 172.21.15.149:0/2763223831 conn(0x7f00f00f1c70 msgr2=0x7f00f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:56.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.932+0000 7f00f5a57700 1 -- 172.21.15.149:0/2763223831 shutdown_connections 2024-02-12T16:37:56.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.932+0000 7f00f5a57700 1 -- 172.21.15.149:0/2763223831 wait complete. 2024-02-12T16:37:56.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.933+0000 7f00f5a57700 1 Processor -- start 2024-02-12T16:37:56.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.933+0000 7f00f5a57700 1 -- start start 2024-02-12T16:37:56.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.933+0000 7f00f5a57700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f00f6510 0x7f00f01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:56.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.933+0000 7f00f5a57700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f0106cd0 con 0x7f00f00f6510 2024-02-12T16:37:56.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.933+0000 7f00eeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f00f6510 0x7f00f01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:56.937 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.933+0000 7f00eeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f00f6510 0x7f00f01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:56000/0 (socket says 172.21.15.149:56000) 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.933+0000 7f00eeffd700 1 -- 172.21.15.149:0/3834695435 learned_addr learned my addr 172.21.15.149:0/3834695435 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.934+0000 7f00eeffd700 1 -- 172.21.15.149:0/3834695435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e4005040 con 0x7f00f00f6510 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.934+0000 7f00eeffd700 1 --2- 172.21.15.149:0/3834695435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f00f6510 0x7f00f01067e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f00e4005010 tx=0x7f00e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.934+0000 7f00d77fe700 1 -- 172.21.15.149:0/3834695435 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00e4009320 con 0x7f00f00f6510 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.934+0000 7f00d77fe700 1 -- 172.21.15.149:0/3834695435 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00e4013380 con 0x7f00f00f6510 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.934+0000 7f00f5a57700 1 -- 172.21.15.149:0/3834695435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f0106ed0 con 0x7f00f00f6510 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.934+0000 7f00d77fe700 1 -- 172.21.15.149:0/3834695435 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00e4014e50 con 0x7f00f00f6510 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.934+0000 7f00f5a57700 1 -- 172.21.15.149:0/3834695435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f0107360 con 0x7f00f00f6510 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.935+0000 7f00d77fe700 1 -- 172.21.15.149:0/3834695435 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00e4014420 con 0x7f00f00f6510 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.935+0000 7f00f5a57700 1 -- 172.21.15.149:0/3834695435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f0045ad0 con 0x7f00f00f6510 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.935+0000 7f00d77fe700 1 --2- 172.21.15.149:0/3834695435 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f00d802ff10 0x7f00d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:37:56.938 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.935+0000 7f00d77fe700 1 -- 172.21.15.149:0/3834695435 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f00e4040e30 con 0x7f00f00f6510 2024-02-12T16:37:56.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.935+0000 7f00ee7fc700 1 --2- 172.21.15.149:0/3834695435 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f00d802ff10 0x7f00d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:37:56.939 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.936+0000 7f00ee7fc700 1 --2- 172.21.15.149:0/3834695435 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f00d802ff10 0x7f00d80323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f00e00096a0 tx=0x7f00e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:37:56.941 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:56.940+0000 7f00d77fe700 1 -- 172.21.15.149:0/3834695435 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00e40095f0 con 0x7f00f00f6510 2024-02-12T16:37:57.144 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.142+0000 7f00f5a57700 1 -- 172.21.15.149:0/3834695435 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00f0108210 con 0x7f00f00f6510 2024-02-12T16:37:57.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.143+0000 7f00d77fe700 1 -- 172.21.15.149:0/3834695435 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f00e401db70 con 0x7f00f00f6510 2024-02-12T16:37:57.145 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:37:57.145 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:37:57.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.145+0000 7f00f5a57700 1 -- 172.21.15.149:0/3834695435 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f00d802ff10 msgr2=0x7f00d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:57.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.145+0000 7f00f5a57700 1 --2- 172.21.15.149:0/3834695435 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f00d802ff10 0x7f00d80323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f00e00096a0 tx=0x7f00e0006b40).stop 2024-02-12T16:37:57.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.145+0000 7f00f5a57700 1 -- 172.21.15.149:0/3834695435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f00f6510 msgr2=0x7f00f01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:37:57.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.145+0000 7f00f5a57700 1 --2- 172.21.15.149:0/3834695435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f00f6510 0x7f00f01067e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f00e4005010 tx=0x7f00e400b040).stop 2024-02-12T16:37:57.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.145+0000 7f00f5a57700 1 -- 172.21.15.149:0/3834695435 shutdown_connections 2024-02-12T16:37:57.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.145+0000 7f00f5a57700 1 --2- 172.21.15.149:0/3834695435 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f00d802ff10 0x7f00d80323d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:57.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.145+0000 7f00f5a57700 1 --2- 172.21.15.149:0/3834695435 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f00f6510 0x7f00f01067e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:37:57.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.145+0000 7f00f5a57700 1 -- 172.21.15.149:0/3834695435 >> 172.21.15.149:0/3834695435 conn(0x7f00f00f1c70 msgr2=0x7f00f00f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:37:57.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.145+0000 7f00f5a57700 1 -- 172.21.15.149:0/3834695435 shutdown_connections 2024-02-12T16:37:57.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:37:57.145+0000 7f00f5a57700 1 -- 172.21.15.149:0/3834695435 wait complete. 2024-02-12T16:37:57.148 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:37:58.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:56.353042+0000 mgr.smithi104.ohwbhf (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:37:58.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:37:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:37:57.144390+0000 mon.smithi104 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.149:0/3834695435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:37:58.797 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:37:58.797 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:00.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:37:58.353387+0000 mgr.smithi104.ohwbhf (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:00.599 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.596+0000 7f21fe443700 1 -- 172.21.15.149:0/3814842657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21f80f6510 msgr2=0x7f21f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:00.599 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.596+0000 7f21fe443700 1 --2- 172.21.15.149:0/3814842657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21f80f6510 0x7f21f80f6930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f21e8009fb0 tx=0x7f21e8009b30).stop 2024-02-12T16:38:00.599 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.596+0000 7f21fe443700 1 -- 172.21.15.149:0/3814842657 shutdown_connections 2024-02-12T16:38:00.599 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.596+0000 7f21fe443700 1 --2- 172.21.15.149:0/3814842657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21f80f6510 0x7f21f80f6930 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:00.600 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.596+0000 7f21fe443700 1 -- 172.21.15.149:0/3814842657 >> 172.21.15.149:0/3814842657 conn(0x7f21f80f1c70 msgr2=0x7f21f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:00.600 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.596+0000 7f21fe443700 1 -- 172.21.15.149:0/3814842657 shutdown_connections 2024-02-12T16:38:00.600 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.596+0000 7f21fe443700 1 -- 172.21.15.149:0/3814842657 wait complete. 2024-02-12T16:38:00.600 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.597+0000 7f21fe443700 1 Processor -- start 2024-02-12T16:38:00.600 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.597+0000 7f21fe443700 1 -- start start 2024-02-12T16:38:00.600 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.597+0000 7f21fe443700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:00.600 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.597+0000 7f21fe443700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f8106d00 con 0x7f21f80f6510 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.597+0000 7f21f7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.598+0000 7f21f7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:56012/0 (socket says 172.21.15.149:56012) 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.598+0000 7f21f7fff700 1 -- 172.21.15.149:0/1929681064 learned_addr learned my addr 172.21.15.149:0/1929681064 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.598+0000 7f21f7fff700 1 -- 172.21.15.149:0/1929681064 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e8005040 con 0x7f21f80f6510 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.598+0000 7f21f7fff700 1 --2- 172.21.15.149:0/1929681064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f21e8005010 tx=0x7f21e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.598+0000 7f21f4ff9700 1 -- 172.21.15.149:0/1929681064 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21e8009320 con 0x7f21f80f6510 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.598+0000 7f21fe443700 1 -- 172.21.15.149:0/1929681064 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f8106f00 con 0x7f21f80f6510 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.598+0000 7f21f4ff9700 1 -- 172.21.15.149:0/1929681064 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21e8013380 con 0x7f21f80f6510 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.598+0000 7f21f4ff9700 1 -- 172.21.15.149:0/1929681064 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21e8014e50 con 0x7f21f80f6510 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.598+0000 7f21fe443700 1 -- 172.21.15.149:0/1929681064 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f8107390 con 0x7f21f80f6510 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.599+0000 7f21f4ff9700 1 -- 172.21.15.149:0/1929681064 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21e80263a0 con 0x7f21f80f6510 2024-02-12T16:38:00.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.599+0000 7f21fe443700 1 -- 172.21.15.149:0/1929681064 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f8045ad0 con 0x7f21f80f6510 2024-02-12T16:38:00.603 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.599+0000 7f21f4ff9700 1 --2- 172.21.15.149:0/1929681064 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21e402ff10 0x7f21e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:00.603 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.599+0000 7f21f4ff9700 1 -- 172.21.15.149:0/1929681064 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21e8041250 con 0x7f21f80f6510 2024-02-12T16:38:00.603 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.599+0000 7f21f77fe700 1 --2- 172.21.15.149:0/1929681064 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21e402ff10 0x7f21e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:00.603 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.600+0000 7f21f77fe700 1 --2- 172.21.15.149:0/1929681064 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21e402ff10 0x7f21e40323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f21e00096a0 tx=0x7f21e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:00.606 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.604+0000 7f21f4ff9700 1 -- 172.21.15.149:0/1929681064 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21e801d470 con 0x7f21f80f6510 2024-02-12T16:38:00.811 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.809+0000 7f21fe443700 1 -- 172.21.15.149:0/1929681064 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21f8108240 con 0x7f21f80f6510 2024-02-12T16:38:00.811 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.810+0000 7f21f4ff9700 1 -- 172.21.15.149:0/1929681064 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f21e801cb10 con 0x7f21f80f6510 2024-02-12T16:38:00.812 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:00.812 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:00.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.811+0000 7f21fe443700 1 -- 172.21.15.149:0/1929681064 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21e402ff10 msgr2=0x7f21e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:00.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.811+0000 7f21fe443700 1 --2- 172.21.15.149:0/1929681064 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21e402ff10 0x7f21e40323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f21e00096a0 tx=0x7f21e0006b40).stop 2024-02-12T16:38:00.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.811+0000 7f21fe443700 1 -- 172.21.15.149:0/1929681064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21f80f6510 msgr2=0x7f21f8106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:00.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.811+0000 7f21fe443700 1 --2- 172.21.15.149:0/1929681064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f21e8005010 tx=0x7f21e800b040).stop 2024-02-12T16:38:00.813 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.811+0000 7f21fe443700 1 -- 172.21.15.149:0/1929681064 shutdown_connections 2024-02-12T16:38:00.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.811+0000 7f21fe443700 1 --2- 172.21.15.149:0/1929681064 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f21e402ff10 0x7f21e40323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:00.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.811+0000 7f21fe443700 1 --2- 172.21.15.149:0/1929681064 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:00.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.811+0000 7f21fe443700 1 -- 172.21.15.149:0/1929681064 >> 172.21.15.149:0/1929681064 conn(0x7f21f80f1c70 msgr2=0x7f21f80f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:00.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.811+0000 7f21fe443700 1 -- 172.21.15.149:0/1929681064 shutdown_connections 2024-02-12T16:38:00.814 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:00.811+0000 7f21fe443700 1 -- 172.21.15.149:0/1929681064 wait complete. 2024-02-12T16:38:00.814 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:01.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:00.810902+0000 mon.smithi104 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.149:0/1929681064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:02.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:00.353692+0000 mgr.smithi104.ohwbhf (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:02.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:02.564 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:04.390 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.386+0000 7fe52f535700 1 -- 172.21.15.149:0/1025954355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5280601c0 msgr2=0x7fe5280f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:04.390 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.386+0000 7fe52f535700 1 --2- 172.21.15.149:0/1025954355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5280601c0 0x7fe5280f91b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fe518009fb0 tx=0x7fe518009b30).stop 2024-02-12T16:38:04.390 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.386+0000 7fe52f535700 1 -- 172.21.15.149:0/1025954355 shutdown_connections 2024-02-12T16:38:04.390 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.386+0000 7fe52f535700 1 --2- 172.21.15.149:0/1025954355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5280601c0 0x7fe5280f91b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:04.390 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.386+0000 7fe52f535700 1 -- 172.21.15.149:0/1025954355 >> 172.21.15.149:0/1025954355 conn(0x7fe5280f0b70 msgr2=0x7fe5280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:04.390 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.386+0000 7fe52f535700 1 -- 172.21.15.149:0/1025954355 shutdown_connections 2024-02-12T16:38:04.390 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.386+0000 7fe52f535700 1 -- 172.21.15.149:0/1025954355 wait complete. 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.386+0000 7fe52f535700 1 Processor -- start 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.387+0000 7fe52f535700 1 -- start start 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.387+0000 7fe52f535700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5280601c0 0x7fe5281089b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.387+0000 7fe52f535700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528108ea0 con 0x7fe5280601c0 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.387+0000 7fe52d2d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5280601c0 0x7fe5281089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.387+0000 7fe52d2d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5280601c0 0x7fe5281089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:42646/0 (socket says 172.21.15.149:42646) 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.387+0000 7fe52d2d1700 1 -- 172.21.15.149:0/979170633 learned_addr learned my addr 172.21.15.149:0/979170633 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.387+0000 7fe52d2d1700 1 -- 172.21.15.149:0/979170633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe518005040 con 0x7fe5280601c0 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.387+0000 7fe52d2d1700 1 --2- 172.21.15.149:0/979170633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5280601c0 0x7fe5281089b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fe518009830 tx=0x7fe51800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.388+0000 7fe51dffb700 1 -- 172.21.15.149:0/979170633 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe518009320 con 0x7fe5280601c0 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.388+0000 7fe52f535700 1 -- 172.21.15.149:0/979170633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5281090a0 con 0x7fe5280601c0 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.388+0000 7fe51dffb700 1 -- 172.21.15.149:0/979170633 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe518013380 con 0x7fe5280601c0 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.388+0000 7fe51dffb700 1 -- 172.21.15.149:0/979170633 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe518014420 con 0x7fe5280601c0 2024-02-12T16:38:04.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.388+0000 7fe52f535700 1 -- 172.21.15.149:0/979170633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe528109530 con 0x7fe5280601c0 2024-02-12T16:38:04.392 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.389+0000 7fe51dffb700 1 -- 172.21.15.149:0/979170633 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe51801e420 con 0x7fe5280601c0 2024-02-12T16:38:04.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.389+0000 7fe52f535700 1 -- 172.21.15.149:0/979170633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe528045ad0 con 0x7fe5280601c0 2024-02-12T16:38:04.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.389+0000 7fe51dffb700 1 --2- 172.21.15.149:0/979170633 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe51402ffb0 0x7fe514032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:04.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.389+0000 7fe51dffb700 1 -- 172.21.15.149:0/979170633 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe518040e70 con 0x7fe5280601c0 2024-02-12T16:38:04.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.389+0000 7fe52cad0700 1 --2- 172.21.15.149:0/979170633 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe51402ffb0 0x7fe514032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:04.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.389+0000 7fe52cad0700 1 --2- 172.21.15.149:0/979170633 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe51402ffb0 0x7fe514032470 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe5240096a0 tx=0x7fe524006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:04.395 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.393+0000 7fe51dffb700 1 -- 172.21.15.149:0/979170633 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5180095f0 con 0x7fe5280601c0 2024-02-12T16:38:04.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:02.353975+0000 mgr.smithi104.ohwbhf (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:04.599 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.597+0000 7fe52f535700 1 -- 172.21.15.149:0/979170633 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5280fc090 con 0x7fe5280601c0 2024-02-12T16:38:04.600 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.598+0000 7fe51dffb700 1 -- 172.21.15.149:0/979170633 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe51801cb10 con 0x7fe5280601c0 2024-02-12T16:38:04.600 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:04.600 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:04.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.600+0000 7fe52f535700 1 -- 172.21.15.149:0/979170633 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe51402ffb0 msgr2=0x7fe514032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:04.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.600+0000 7fe52f535700 1 --2- 172.21.15.149:0/979170633 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe51402ffb0 0x7fe514032470 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe5240096a0 tx=0x7fe524006b40).stop 2024-02-12T16:38:04.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.600+0000 7fe52f535700 1 -- 172.21.15.149:0/979170633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5280601c0 msgr2=0x7fe5281089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:04.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.600+0000 7fe52f535700 1 --2- 172.21.15.149:0/979170633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5280601c0 0x7fe5281089b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fe518009830 tx=0x7fe51800b040).stop 2024-02-12T16:38:04.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.600+0000 7fe52f535700 1 -- 172.21.15.149:0/979170633 shutdown_connections 2024-02-12T16:38:04.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.600+0000 7fe52f535700 1 --2- 172.21.15.149:0/979170633 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fe51402ffb0 0x7fe514032470 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:04.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.600+0000 7fe52f535700 1 --2- 172.21.15.149:0/979170633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5280601c0 0x7fe5281089b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:04.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.600+0000 7fe52f535700 1 -- 172.21.15.149:0/979170633 >> 172.21.15.149:0/979170633 conn(0x7fe5280f0b70 msgr2=0x7fe5280f15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:04.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.600+0000 7fe52f535700 1 -- 172.21.15.149:0/979170633 shutdown_connections 2024-02-12T16:38:04.602 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:04.600+0000 7fe52f535700 1 -- 172.21.15.149:0/979170633 wait complete. 2024-02-12T16:38:04.603 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:05.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:04.599410+0000 mon.smithi104 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.149:0/979170633' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:06.302 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:06.302 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:06.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:04.354311+0000 mgr.smithi104.ohwbhf (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:08.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.177+0000 7f792d5e8700 1 -- 172.21.15.149:0/3466213151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79280f6510 msgr2=0x7f79280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:08.179 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.177+0000 7f792d5e8700 1 --2- 172.21.15.149:0/3466213151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79280f6510 0x7f79280f6930 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f791c009fb0 tx=0x7f791c009b30).stop 2024-02-12T16:38:08.180 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.177+0000 7f792d5e8700 1 -- 172.21.15.149:0/3466213151 shutdown_connections 2024-02-12T16:38:08.180 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.177+0000 7f792d5e8700 1 --2- 172.21.15.149:0/3466213151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79280f6510 0x7f79280f6930 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:08.180 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.177+0000 7f792d5e8700 1 -- 172.21.15.149:0/3466213151 >> 172.21.15.149:0/3466213151 conn(0x7f79280f1c70 msgr2=0x7f79280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:08.180 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.177+0000 7f792d5e8700 1 -- 172.21.15.149:0/3466213151 shutdown_connections 2024-02-12T16:38:08.180 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.177+0000 7f792d5e8700 1 -- 172.21.15.149:0/3466213151 wait complete. 2024-02-12T16:38:08.180 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.177+0000 7f792d5e8700 1 Processor -- start 2024-02-12T16:38:08.180 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.177+0000 7f792d5e8700 1 -- start start 2024-02-12T16:38:08.180 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.177+0000 7f792d5e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:08.180 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.178+0000 7f792d5e8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7928069e20 con 0x7f79280f6510 2024-02-12T16:38:08.182 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.178+0000 7f7926ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:08.182 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.178+0000 7f7926ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:42652/0 (socket says 172.21.15.149:42652) 2024-02-12T16:38:08.182 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.178+0000 7f7926ffd700 1 -- 172.21.15.149:0/2610748188 learned_addr learned my addr 172.21.15.149:0/2610748188 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:08.182 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.178+0000 7f7926ffd700 1 -- 172.21.15.149:0/2610748188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c005040 con 0x7f79280f6510 2024-02-12T16:38:08.182 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.178+0000 7f7926ffd700 1 --2- 172.21.15.149:0/2610748188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f791c005010 tx=0x7f791c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:08.182 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.178+0000 7f790f7fe700 1 -- 172.21.15.149:0/2610748188 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f791c009320 con 0x7f79280f6510 2024-02-12T16:38:08.182 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.178+0000 7f790f7fe700 1 -- 172.21.15.149:0/2610748188 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f791c013380 con 0x7f79280f6510 2024-02-12T16:38:08.182 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.178+0000 7f792d5e8700 1 -- 172.21.15.149:0/2610748188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7928069fc0 con 0x7f79280f6510 2024-02-12T16:38:08.183 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.179+0000 7f790f7fe700 1 -- 172.21.15.149:0/2610748188 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f791c014e50 con 0x7f79280f6510 2024-02-12T16:38:08.183 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.179+0000 7f792d5e8700 1 -- 172.21.15.149:0/2610748188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792806a450 con 0x7f79280f6510 2024-02-12T16:38:08.183 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.179+0000 7f790f7fe700 1 -- 172.21.15.149:0/2610748188 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f791c0095f0 con 0x7f79280f6510 2024-02-12T16:38:08.183 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.179+0000 7f792d5e8700 1 -- 172.21.15.149:0/2610748188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7928059100 con 0x7f79280f6510 2024-02-12T16:38:08.183 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.179+0000 7f790f7fe700 1 --2- 172.21.15.149:0/2610748188 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f791002ff10 0x7f79100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:08.183 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.179+0000 7f790f7fe700 1 -- 172.21.15.149:0/2610748188 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f791c040fa0 con 0x7f79280f6510 2024-02-12T16:38:08.183 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.180+0000 7f79267fc700 1 --2- 172.21.15.149:0/2610748188 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f791002ff10 0x7f79100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:08.183 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.180+0000 7f79267fc700 1 --2- 172.21.15.149:0/2610748188 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f791002ff10 0x7f79100323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f79180096a0 tx=0x7f7918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:08.186 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.184+0000 7f790f7fe700 1 -- 172.21.15.149:0/2610748188 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f791c014420 con 0x7f79280f6510 2024-02-12T16:38:08.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.389+0000 7f792d5e8700 1 -- 172.21.15.149:0/2610748188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79280fc0a0 con 0x7f79280f6510 2024-02-12T16:38:08.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.389+0000 7f790f7fe700 1 -- 172.21.15.149:0/2610748188 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f791c010230 con 0x7f79280f6510 2024-02-12T16:38:08.391 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:08.392 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:08.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.391+0000 7f792d5e8700 1 -- 172.21.15.149:0/2610748188 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f791002ff10 msgr2=0x7f79100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:08.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.391+0000 7f792d5e8700 1 --2- 172.21.15.149:0/2610748188 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f791002ff10 0x7f79100323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f79180096a0 tx=0x7f7918006b40).stop 2024-02-12T16:38:08.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.391+0000 7f792d5e8700 1 -- 172.21.15.149:0/2610748188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79280f6510 msgr2=0x7f792806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:08.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.391+0000 7f792d5e8700 1 --2- 172.21.15.149:0/2610748188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f791c005010 tx=0x7f791c00b040).stop 2024-02-12T16:38:08.394 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.391+0000 7f792d5e8700 1 -- 172.21.15.149:0/2610748188 shutdown_connections 2024-02-12T16:38:08.394 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.391+0000 7f792d5e8700 1 --2- 172.21.15.149:0/2610748188 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f791002ff10 0x7f79100323d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:08.394 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.391+0000 7f792d5e8700 1 --2- 172.21.15.149:0/2610748188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:08.394 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.391+0000 7f792d5e8700 1 -- 172.21.15.149:0/2610748188 >> 172.21.15.149:0/2610748188 conn(0x7f79280f1c70 msgr2=0x7f79280f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:08.394 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.391+0000 7f792d5e8700 1 -- 172.21.15.149:0/2610748188 shutdown_connections 2024-02-12T16:38:08.394 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:08.391+0000 7f792d5e8700 1 -- 172.21.15.149:0/2610748188 wait complete. 2024-02-12T16:38:08.394 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:08.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:06.354711+0000 mgr.smithi104.ohwbhf (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:09.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:08.390603+0000 mon.smithi104 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.149:0/2610748188' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:10.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:10.137 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:10.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:08.355031+0000 mgr.smithi104.ohwbhf (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:12.032 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.030+0000 7fab40306700 1 -- 172.21.15.149:0/2796785785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab380f5410 msgr2=0x7fab380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:12.033 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.030+0000 7fab40306700 1 --2- 172.21.15.149:0/2796785785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab380f5410 0x7fab380f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fab2c009fb0 tx=0x7fab2c009b30).stop 2024-02-12T16:38:12.033 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.030+0000 7fab40306700 1 -- 172.21.15.149:0/2796785785 shutdown_connections 2024-02-12T16:38:12.033 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.030+0000 7fab40306700 1 --2- 172.21.15.149:0/2796785785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab380f5410 0x7fab380f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:12.033 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.030+0000 7fab40306700 1 -- 172.21.15.149:0/2796785785 >> 172.21.15.149:0/2796785785 conn(0x7fab380f0b70 msgr2=0x7fab380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:12.033 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.030+0000 7fab40306700 1 -- 172.21.15.149:0/2796785785 shutdown_connections 2024-02-12T16:38:12.033 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.030+0000 7fab40306700 1 -- 172.21.15.149:0/2796785785 wait complete. 2024-02-12T16:38:12.035 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.031+0000 7fab40306700 1 Processor -- start 2024-02-12T16:38:12.035 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.031+0000 7fab40306700 1 -- start start 2024-02-12T16:38:12.035 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.031+0000 7fab40306700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab380f5410 0x7fab38105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:12.035 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.031+0000 7fab40306700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab38105c10 con 0x7fab380f5410 2024-02-12T16:38:12.035 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.031+0000 7fab3e0a2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab380f5410 0x7fab38105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:12.035 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.031+0000 7fab3e0a2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab380f5410 0x7fab38105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:42670/0 (socket says 172.21.15.149:42670) 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.031+0000 7fab3e0a2700 1 -- 172.21.15.149:0/3975709708 learned_addr learned my addr 172.21.15.149:0/3975709708 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.032+0000 7fab3e0a2700 1 -- 172.21.15.149:0/3975709708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c005040 con 0x7fab380f5410 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.032+0000 7fab3e0a2700 1 --2- 172.21.15.149:0/3975709708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab380f5410 0x7fab38105720 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fab2c005010 tx=0x7fab2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.032+0000 7fab2affd700 1 -- 172.21.15.149:0/3975709708 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2c009320 con 0x7fab380f5410 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.032+0000 7fab40306700 1 -- 172.21.15.149:0/3975709708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab38105e10 con 0x7fab380f5410 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.032+0000 7fab2affd700 1 -- 172.21.15.149:0/3975709708 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab2c013380 con 0x7fab380f5410 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.032+0000 7fab2affd700 1 -- 172.21.15.149:0/3975709708 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2c014e50 con 0x7fab380f5410 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.032+0000 7fab40306700 1 -- 172.21.15.149:0/3975709708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab381062a0 con 0x7fab380f5410 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.033+0000 7fab40306700 1 -- 172.21.15.149:0/3975709708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab38045ad0 con 0x7fab380f5410 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.033+0000 7fab2affd700 1 -- 172.21.15.149:0/3975709708 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab2c0095f0 con 0x7fab380f5410 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.033+0000 7fab2affd700 1 --2- 172.21.15.149:0/3975709708 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab2402ff10 0x7fab240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.033+0000 7fab2affd700 1 -- 172.21.15.149:0/3975709708 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab2c040fa0 con 0x7fab380f5410 2024-02-12T16:38:12.036 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.033+0000 7fab3d8a1700 1 --2- 172.21.15.149:0/3975709708 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab2402ff10 0x7fab240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:12.037 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.034+0000 7fab3d8a1700 1 --2- 172.21.15.149:0/3975709708 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab2402ff10 0x7fab240323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fab340096a0 tx=0x7fab34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:12.039 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.037+0000 7fab2affd700 1 -- 172.21.15.149:0/3975709708 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab2c014420 con 0x7fab380f5410 2024-02-12T16:38:12.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.242+0000 7fab40306700 1 -- 172.21.15.149:0/3975709708 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab38107150 con 0x7fab380f5410 2024-02-12T16:38:12.244 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.243+0000 7fab2affd700 1 -- 172.21.15.149:0/3975709708 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fab2c010230 con 0x7fab380f5410 2024-02-12T16:38:12.245 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:12.245 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:12.246 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.244+0000 7fab40306700 1 -- 172.21.15.149:0/3975709708 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab2402ff10 msgr2=0x7fab240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:12.246 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.244+0000 7fab40306700 1 --2- 172.21.15.149:0/3975709708 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab2402ff10 0x7fab240323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fab340096a0 tx=0x7fab34006b40).stop 2024-02-12T16:38:12.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.244+0000 7fab40306700 1 -- 172.21.15.149:0/3975709708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab380f5410 msgr2=0x7fab38105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:12.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.244+0000 7fab40306700 1 --2- 172.21.15.149:0/3975709708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab380f5410 0x7fab38105720 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fab2c005010 tx=0x7fab2c00b040).stop 2024-02-12T16:38:12.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.244+0000 7fab40306700 1 -- 172.21.15.149:0/3975709708 shutdown_connections 2024-02-12T16:38:12.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.244+0000 7fab40306700 1 --2- 172.21.15.149:0/3975709708 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab2402ff10 0x7fab240323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:12.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.244+0000 7fab40306700 1 --2- 172.21.15.149:0/3975709708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab380f5410 0x7fab38105720 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:12.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.244+0000 7fab40306700 1 -- 172.21.15.149:0/3975709708 >> 172.21.15.149:0/3975709708 conn(0x7fab380f0b70 msgr2=0x7fab380f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:12.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.244+0000 7fab40306700 1 -- 172.21.15.149:0/3975709708 shutdown_connections 2024-02-12T16:38:12.247 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:12.244+0000 7fab40306700 1 -- 172.21.15.149:0/3975709708 wait complete. 2024-02-12T16:38:12.247 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:12.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:10.355437+0000 mgr.smithi104.ohwbhf (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:13.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:12.243817+0000 mon.smithi104 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.149:0/3975709708' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:13.833 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:13.833 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:14.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:12.355701+0000 mgr.smithi104.ohwbhf (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:15.743 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.740+0000 7f8888e89700 1 -- 172.21.15.149:0/1534299786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88840f51a0 msgr2=0x7f88840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:15.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.740+0000 7f8888e89700 1 --2- 172.21.15.149:0/1534299786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88840f51a0 0x7f88840f55c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f8874009fb0 tx=0x7f8874009b30).stop 2024-02-12T16:38:15.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.740+0000 7f8888e89700 1 -- 172.21.15.149:0/1534299786 shutdown_connections 2024-02-12T16:38:15.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.740+0000 7f8888e89700 1 --2- 172.21.15.149:0/1534299786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88840f51a0 0x7f88840f55c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:15.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.740+0000 7f8888e89700 1 -- 172.21.15.149:0/1534299786 >> 172.21.15.149:0/1534299786 conn(0x7f88840f09a0 msgr2=0x7f88840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:15.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.740+0000 7f8888e89700 1 -- 172.21.15.149:0/1534299786 shutdown_connections 2024-02-12T16:38:15.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.740+0000 7f8888e89700 1 -- 172.21.15.149:0/1534299786 wait complete. 2024-02-12T16:38:15.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.740+0000 7f8888e89700 1 Processor -- start 2024-02-12T16:38:15.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.740+0000 7f8888e89700 1 -- start start 2024-02-12T16:38:15.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f8888e89700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88840f51a0 0x7f88840fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:15.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f8888e89700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88840fb6f0 con 0x7f88840f51a0 2024-02-12T16:38:15.744 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f88837fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88840f51a0 0x7f88840fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:15.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f88837fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88840f51a0 0x7f88840fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55400/0 (socket says 172.21.15.149:55400) 2024-02-12T16:38:15.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f88837fe700 1 -- 172.21.15.149:0/4031867403 learned_addr learned my addr 172.21.15.149:0/4031867403 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:15.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f88837fe700 1 -- 172.21.15.149:0/4031867403 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8874005040 con 0x7f88840f51a0 2024-02-12T16:38:15.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f88837fe700 1 --2- 172.21.15.149:0/4031867403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88840f51a0 0x7f88840fd270 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f88740089f0 tx=0x7f887400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:15.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f8880ff9700 1 -- 172.21.15.149:0/4031867403 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8874009320 con 0x7f88840f51a0 2024-02-12T16:38:15.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f8888e89700 1 -- 172.21.15.149:0/4031867403 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88840fb890 con 0x7f88840f51a0 2024-02-12T16:38:15.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f8880ff9700 1 -- 172.21.15.149:0/4031867403 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8874013380 con 0x7f88840f51a0 2024-02-12T16:38:15.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f8880ff9700 1 -- 172.21.15.149:0/4031867403 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8874014e20 con 0x7f88840f51a0 2024-02-12T16:38:15.745 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.741+0000 7f8888e89700 1 -- 172.21.15.149:0/4031867403 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88840fbd20 con 0x7f88840f51a0 2024-02-12T16:38:15.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.742+0000 7f8880ff9700 1 -- 172.21.15.149:0/4031867403 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88740095f0 con 0x7f88840f51a0 2024-02-12T16:38:15.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.742+0000 7f8888e89700 1 -- 172.21.15.149:0/4031867403 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8884059100 con 0x7f88840f51a0 2024-02-12T16:38:15.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.742+0000 7f8880ff9700 1 --2- 172.21.15.149:0/4031867403 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f887002ff10 0x7f88700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:15.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.742+0000 7f8880ff9700 1 -- 172.21.15.149:0/4031867403 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8874040fb0 con 0x7f88840f51a0 2024-02-12T16:38:15.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.743+0000 7f887bfff700 1 --2- 172.21.15.149:0/4031867403 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f887002ff10 0x7f88700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:15.746 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.743+0000 7f887bfff700 1 --2- 172.21.15.149:0/4031867403 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f887002ff10 0x7f88700323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f886c0096a0 tx=0x7f886c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:15.749 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.747+0000 7f8880ff9700 1 -- 172.21.15.149:0/4031867403 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8874010230 con 0x7f88840f51a0 2024-02-12T16:38:15.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.950+0000 7f8888e89700 1 -- 172.21.15.149:0/4031867403 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8884045ad0 con 0x7f88840f51a0 2024-02-12T16:38:15.953 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.951+0000 7f8880ff9700 1 -- 172.21.15.149:0/4031867403 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8874010230 con 0x7f88840f51a0 2024-02-12T16:38:15.954 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:15.954 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:15.955 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.952+0000 7f8888e89700 1 -- 172.21.15.149:0/4031867403 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f887002ff10 msgr2=0x7f88700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:15.955 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.952+0000 7f8888e89700 1 --2- 172.21.15.149:0/4031867403 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f887002ff10 0x7f88700323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f886c0096a0 tx=0x7f886c006b40).stop 2024-02-12T16:38:15.955 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.952+0000 7f8888e89700 1 -- 172.21.15.149:0/4031867403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88840f51a0 msgr2=0x7f88840fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:15.955 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.952+0000 7f8888e89700 1 --2- 172.21.15.149:0/4031867403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88840f51a0 0x7f88840fd270 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f88740089f0 tx=0x7f887400b040).stop 2024-02-12T16:38:15.955 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.953+0000 7f8888e89700 1 -- 172.21.15.149:0/4031867403 shutdown_connections 2024-02-12T16:38:15.955 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.953+0000 7f8888e89700 1 --2- 172.21.15.149:0/4031867403 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f887002ff10 0x7f88700323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:15.955 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.953+0000 7f8888e89700 1 --2- 172.21.15.149:0/4031867403 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88840f51a0 0x7f88840fd270 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:15.955 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.953+0000 7f8888e89700 1 -- 172.21.15.149:0/4031867403 >> 172.21.15.149:0/4031867403 conn(0x7f88840f09a0 msgr2=0x7f888405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:15.955 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.953+0000 7f8888e89700 1 -- 172.21.15.149:0/4031867403 shutdown_connections 2024-02-12T16:38:15.955 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:15.953+0000 7f8888e89700 1 -- 172.21.15.149:0/4031867403 wait complete. 2024-02-12T16:38:15.956 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:16.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:14.356086+0000 mgr.smithi104.ohwbhf (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:16.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:15.952485+0000 mon.smithi104 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.149:0/4031867403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:17.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:17.735 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:18.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:16.356547+0000 mgr.smithi104.ohwbhf (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:19.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.558+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2656550597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9a40f51a0 msgr2=0x7fb9a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:19.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.558+0000 7fb9aa4f0700 1 --2- 172.21.15.149:0/2656550597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9a40f51a0 0x7fb9a40f55c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fb9a0009fb0 tx=0x7fb9a0009b30).stop 2024-02-12T16:38:19.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.558+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2656550597 shutdown_connections 2024-02-12T16:38:19.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.558+0000 7fb9aa4f0700 1 --2- 172.21.15.149:0/2656550597 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9a40f51a0 0x7fb9a40f55c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:19.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.558+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2656550597 >> 172.21.15.149:0/2656550597 conn(0x7fb9a40f09a0 msgr2=0x7fb9a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:19.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.558+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2656550597 shutdown_connections 2024-02-12T16:38:19.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.558+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2656550597 wait complete. 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.558+0000 7fb9aa4f0700 1 Processor -- start 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.559+0000 7fb9aa4f0700 1 -- start start 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.559+0000 7fb9aa4f0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9a40f51a0 0x7fb9a4109280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.559+0000 7fb9aa4f0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a4060280 con 0x7fb9a40f51a0 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.559+0000 7fb9a94ee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9a40f51a0 0x7fb9a4109280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.559+0000 7fb9a94ee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9a40f51a0 0x7fb9a4109280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55418/0 (socket says 172.21.15.149:55418) 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.559+0000 7fb9a94ee700 1 -- 172.21.15.149:0/2470889442 learned_addr learned my addr 172.21.15.149:0/2470889442 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.559+0000 7fb9a94ee700 1 -- 172.21.15.149:0/2470889442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a0005040 con 0x7fb9a40f51a0 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.559+0000 7fb9a94ee700 1 --2- 172.21.15.149:0/2470889442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9a40f51a0 0x7fb9a4109280 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fb9a00089f0 tx=0x7fb9a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.560+0000 7fb999ffb700 1 -- 172.21.15.149:0/2470889442 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9a0009320 con 0x7fb9a40f51a0 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.560+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2470889442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a4060480 con 0x7fb9a40f51a0 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.560+0000 7fb999ffb700 1 -- 172.21.15.149:0/2470889442 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9a0013380 con 0x7fb9a40f51a0 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.560+0000 7fb999ffb700 1 -- 172.21.15.149:0/2470889442 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9a0014e20 con 0x7fb9a40f51a0 2024-02-12T16:38:19.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.560+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2470889442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a41099b0 con 0x7fb9a40f51a0 2024-02-12T16:38:19.565 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.561+0000 7fb999ffb700 1 -- 172.21.15.149:0/2470889442 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9a00263a0 con 0x7fb9a40f51a0 2024-02-12T16:38:19.565 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.561+0000 7fb999ffb700 1 --2- 172.21.15.149:0/2470889442 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb99002ff10 0x7fb9900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:19.565 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.561+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2470889442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a4045ad0 con 0x7fb9a40f51a0 2024-02-12T16:38:19.565 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.561+0000 7fb999ffb700 1 -- 172.21.15.149:0/2470889442 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9a0041190 con 0x7fb9a40f51a0 2024-02-12T16:38:19.565 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.561+0000 7fb9a8ced700 1 --2- 172.21.15.149:0/2470889442 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb99002ff10 0x7fb9900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:19.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.561+0000 7fb9a8ced700 1 --2- 172.21.15.149:0/2470889442 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb99002ff10 0x7fb9900323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fb9940096a0 tx=0x7fb994006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:19.567 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.565+0000 7fb999ffb700 1 -- 172.21.15.149:0/2470889442 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9a0012330 con 0x7fb9a40f51a0 2024-02-12T16:38:19.770 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.768+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2470889442 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9a40fad30 con 0x7fb9a40f51a0 2024-02-12T16:38:19.771 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.769+0000 7fb999ffb700 1 -- 172.21.15.149:0/2470889442 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb9a0012330 con 0x7fb9a40f51a0 2024-02-12T16:38:19.771 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:19.771 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:19.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.771+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2470889442 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb99002ff10 msgr2=0x7fb9900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:19.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.771+0000 7fb9aa4f0700 1 --2- 172.21.15.149:0/2470889442 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb99002ff10 0x7fb9900323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fb9940096a0 tx=0x7fb994006b40).stop 2024-02-12T16:38:19.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.771+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2470889442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9a40f51a0 msgr2=0x7fb9a4109280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:19.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.771+0000 7fb9aa4f0700 1 --2- 172.21.15.149:0/2470889442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9a40f51a0 0x7fb9a4109280 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fb9a00089f0 tx=0x7fb9a000b040).stop 2024-02-12T16:38:19.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.771+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2470889442 shutdown_connections 2024-02-12T16:38:19.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.771+0000 7fb9aa4f0700 1 --2- 172.21.15.149:0/2470889442 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb99002ff10 0x7fb9900323d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:19.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.771+0000 7fb9aa4f0700 1 --2- 172.21.15.149:0/2470889442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9a40f51a0 0x7fb9a4109280 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:19.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.771+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2470889442 >> 172.21.15.149:0/2470889442 conn(0x7fb9a40f09a0 msgr2=0x7fb9a40f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:19.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.771+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2470889442 shutdown_connections 2024-02-12T16:38:19.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:19.771+0000 7fb9aa4f0700 1 -- 172.21.15.149:0/2470889442 wait complete. 2024-02-12T16:38:19.774 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:20.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:18.357018+0000 mgr.smithi104.ohwbhf (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:20.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:19.770361+0000 mon.smithi104 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.149:0/2470889442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:21.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:21.518 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:22.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:20.357457+0000 mgr.smithi104.ohwbhf (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:23.368 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.351+0000 7efffca83700 1 -- 172.21.15.149:0/1870440339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efff80f51a0 msgr2=0x7efff80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.351+0000 7efffca83700 1 --2- 172.21.15.149:0/1870440339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efff80f51a0 0x7efff80f55c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7effe8009fb0 tx=0x7effe8009b30).stop 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.351+0000 7efffca83700 1 -- 172.21.15.149:0/1870440339 shutdown_connections 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.351+0000 7efffca83700 1 --2- 172.21.15.149:0/1870440339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efff80f51a0 0x7efff80f55c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.351+0000 7efffca83700 1 -- 172.21.15.149:0/1870440339 >> 172.21.15.149:0/1870440339 conn(0x7efff80f09a0 msgr2=0x7efff80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.351+0000 7efffca83700 1 -- 172.21.15.149:0/1870440339 shutdown_connections 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.351+0000 7efffca83700 1 -- 172.21.15.149:0/1870440339 wait complete. 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.351+0000 7efffca83700 1 Processor -- start 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.352+0000 7efffca83700 1 -- start start 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.352+0000 7efffca83700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efff80f51a0 0x7efff81092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.352+0000 7efffca83700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff8060220 con 0x7efff80f51a0 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.352+0000 7efff77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efff80f51a0 0x7efff81092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:23.369 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.352+0000 7efff77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efff80f51a0 0x7efff81092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:55434/0 (socket says 172.21.15.149:55434) 2024-02-12T16:38:23.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.352+0000 7efff77fe700 1 -- 172.21.15.149:0/3093727126 learned_addr learned my addr 172.21.15.149:0/3093727126 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:23.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.352+0000 7efff77fe700 1 -- 172.21.15.149:0/3093727126 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe8005040 con 0x7efff80f51a0 2024-02-12T16:38:23.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.352+0000 7efff77fe700 1 --2- 172.21.15.149:0/3093727126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efff80f51a0 0x7efff81092f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7effe8009830 tx=0x7effe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:23.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.353+0000 7effdffff700 1 -- 172.21.15.149:0/3093727126 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effe8009320 con 0x7efff80f51a0 2024-02-12T16:38:23.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.353+0000 7effdffff700 1 -- 172.21.15.149:0/3093727126 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effe8013380 con 0x7efff80f51a0 2024-02-12T16:38:23.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.353+0000 7effdffff700 1 -- 172.21.15.149:0/3093727126 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effe8014420 con 0x7efff80f51a0 2024-02-12T16:38:23.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.353+0000 7efffca83700 1 -- 172.21.15.149:0/3093727126 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff8060420 con 0x7efff80f51a0 2024-02-12T16:38:23.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.353+0000 7efffca83700 1 -- 172.21.15.149:0/3093727126 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff81099a0 con 0x7efff80f51a0 2024-02-12T16:38:23.370 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.353+0000 7effdffff700 1 -- 172.21.15.149:0/3093727126 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effe801e420 con 0x7efff80f51a0 2024-02-12T16:38:23.371 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.354+0000 7efffca83700 1 -- 172.21.15.149:0/3093727126 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff8059100 con 0x7efff80f51a0 2024-02-12T16:38:23.371 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.354+0000 7effdffff700 1 --2- 172.21.15.149:0/3093727126 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7effe002ffb0 0x7effe0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:23.371 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.354+0000 7effdffff700 1 -- 172.21.15.149:0/3093727126 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effe8040eb0 con 0x7efff80f51a0 2024-02-12T16:38:23.371 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.354+0000 7efff6ffd700 1 --2- 172.21.15.149:0/3093727126 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7effe002ffb0 0x7effe0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:23.371 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.354+0000 7efff6ffd700 1 --2- 172.21.15.149:0/3093727126 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7effe002ffb0 0x7effe0032470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7effec0096a0 tx=0x7effec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:23.371 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.358+0000 7effdffff700 1 -- 172.21.15.149:0/3093727126 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effe80095f0 con 0x7efff80f51a0 2024-02-12T16:38:23.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.564+0000 7efffca83700 1 -- 172.21.15.149:0/3093727126 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efff8045ad0 con 0x7efff80f51a0 2024-02-12T16:38:23.567 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.565+0000 7effdffff700 1 -- 172.21.15.149:0/3093727126 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7effe801cb10 con 0x7efff80f51a0 2024-02-12T16:38:23.567 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:23.567 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:23.568 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.566+0000 7efffca83700 1 -- 172.21.15.149:0/3093727126 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7effe002ffb0 msgr2=0x7effe0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:23.568 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.566+0000 7efffca83700 1 --2- 172.21.15.149:0/3093727126 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7effe002ffb0 0x7effe0032470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7effec0096a0 tx=0x7effec006b40).stop 2024-02-12T16:38:23.568 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.566+0000 7efffca83700 1 -- 172.21.15.149:0/3093727126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efff80f51a0 msgr2=0x7efff81092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:23.568 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.566+0000 7efffca83700 1 --2- 172.21.15.149:0/3093727126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efff80f51a0 0x7efff81092f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7effe8009830 tx=0x7effe800b040).stop 2024-02-12T16:38:23.568 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.566+0000 7efffca83700 1 -- 172.21.15.149:0/3093727126 shutdown_connections 2024-02-12T16:38:23.568 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.566+0000 7efffca83700 1 --2- 172.21.15.149:0/3093727126 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7effe002ffb0 0x7effe0032470 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:23.569 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.566+0000 7efffca83700 1 --2- 172.21.15.149:0/3093727126 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efff80f51a0 0x7efff81092f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:23.569 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.566+0000 7efffca83700 1 -- 172.21.15.149:0/3093727126 >> 172.21.15.149:0/3093727126 conn(0x7efff80f09a0 msgr2=0x7efff80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:23.569 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.566+0000 7efffca83700 1 -- 172.21.15.149:0/3093727126 shutdown_connections 2024-02-12T16:38:23.569 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:23.566+0000 7efffca83700 1 -- 172.21.15.149:0/3093727126 wait complete. 2024-02-12T16:38:23.569 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:24.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:22.357735+0000 mgr.smithi104.ohwbhf (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:24.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:23.565824+0000 mon.smithi104 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.149:0/3093727126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:25.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:25.215 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:24.411728+0000 mon.smithi104 (mon.0) 278 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:38:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:24.417258+0000 mon.smithi104 (mon.0) 279 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:38:26.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:24.358110+0000 mgr.smithi104.ohwbhf (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:27.012 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.007+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/550110019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc540f6510 msgr2=0x7fcc540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:27.012 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.007+0000 7fcc5bb0c700 1 --2- 172.21.15.149:0/550110019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc540f6510 0x7fcc540f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fcc44009fb0 tx=0x7fcc44009b30).stop 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.007+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/550110019 shutdown_connections 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.007+0000 7fcc5bb0c700 1 --2- 172.21.15.149:0/550110019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc540f6510 0x7fcc540f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.007+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/550110019 >> 172.21.15.149:0/550110019 conn(0x7fcc540f1c70 msgr2=0x7fcc540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.007+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/550110019 shutdown_connections 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.008+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/550110019 wait complete. 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.008+0000 7fcc5bb0c700 1 Processor -- start 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.008+0000 7fcc5bb0c700 1 -- start start 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.008+0000 7fcc5bb0c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc540f6510 0x7fcc541067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.008+0000 7fcc5bb0c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54106cd0 con 0x7fcc540f6510 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.009+0000 7fcc598a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc540f6510 0x7fcc541067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.009+0000 7fcc598a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc540f6510 0x7fcc541067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:48092/0 (socket says 172.21.15.149:48092) 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.009+0000 7fcc598a8700 1 -- 172.21.15.149:0/1321945668 learned_addr learned my addr 172.21.15.149:0/1321945668 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:27.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.009+0000 7fcc598a8700 1 -- 172.21.15.149:0/1321945668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc44005040 con 0x7fcc540f6510 2024-02-12T16:38:27.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.009+0000 7fcc598a8700 1 --2- 172.21.15.149:0/1321945668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc540f6510 0x7fcc541067e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fcc44005010 tx=0x7fcc4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:27.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.009+0000 7fcc4a7fc700 1 -- 172.21.15.149:0/1321945668 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc44009320 con 0x7fcc540f6510 2024-02-12T16:38:27.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.009+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/1321945668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc54106ed0 con 0x7fcc540f6510 2024-02-12T16:38:27.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.009+0000 7fcc4a7fc700 1 -- 172.21.15.149:0/1321945668 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc44013380 con 0x7fcc540f6510 2024-02-12T16:38:27.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.009+0000 7fcc4a7fc700 1 -- 172.21.15.149:0/1321945668 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc44014e50 con 0x7fcc540f6510 2024-02-12T16:38:27.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.009+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/1321945668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc54107360 con 0x7fcc540f6510 2024-02-12T16:38:27.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.010+0000 7fcc4a7fc700 1 -- 172.21.15.149:0/1321945668 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc440095f0 con 0x7fcc540f6510 2024-02-12T16:38:27.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.010+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/1321945668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc54045ad0 con 0x7fcc540f6510 2024-02-12T16:38:27.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.010+0000 7fcc4a7fc700 1 --2- 172.21.15.149:0/1321945668 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcc4002ff10 0x7fcc400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:27.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.010+0000 7fcc4a7fc700 1 -- 172.21.15.149:0/1321945668 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc44040e90 con 0x7fcc540f6510 2024-02-12T16:38:27.014 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.010+0000 7fcc590a7700 1 --2- 172.21.15.149:0/1321945668 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcc4002ff10 0x7fcc400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:27.015 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.011+0000 7fcc590a7700 1 --2- 172.21.15.149:0/1321945668 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcc4002ff10 0x7fcc400323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcc500096a0 tx=0x7fcc50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:27.016 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.015+0000 7fcc4a7fc700 1 -- 172.21.15.149:0/1321945668 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc44014420 con 0x7fcc540f6510 2024-02-12T16:38:27.221 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.219+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/1321945668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc54108210 con 0x7fcc540f6510 2024-02-12T16:38:27.222 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.220+0000 7fcc4a7fc700 1 -- 172.21.15.149:0/1321945668 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcc44010230 con 0x7fcc540f6510 2024-02-12T16:38:27.222 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:27.222 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:27.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.221+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/1321945668 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcc4002ff10 msgr2=0x7fcc400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:27.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.221+0000 7fcc5bb0c700 1 --2- 172.21.15.149:0/1321945668 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcc4002ff10 0x7fcc400323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcc500096a0 tx=0x7fcc50006b40).stop 2024-02-12T16:38:27.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.221+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/1321945668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc540f6510 msgr2=0x7fcc541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:27.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.221+0000 7fcc5bb0c700 1 --2- 172.21.15.149:0/1321945668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc540f6510 0x7fcc541067e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fcc44005010 tx=0x7fcc4400b040).stop 2024-02-12T16:38:27.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.222+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/1321945668 shutdown_connections 2024-02-12T16:38:27.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.222+0000 7fcc5bb0c700 1 --2- 172.21.15.149:0/1321945668 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcc4002ff10 0x7fcc400323d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:27.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.222+0000 7fcc5bb0c700 1 --2- 172.21.15.149:0/1321945668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc540f6510 0x7fcc541067e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:27.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.222+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/1321945668 >> 172.21.15.149:0/1321945668 conn(0x7fcc540f1c70 msgr2=0x7fcc540f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:27.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.222+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/1321945668 shutdown_connections 2024-02-12T16:38:27.224 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:27.222+0000 7fcc5bb0c700 1 -- 172.21.15.149:0/1321945668 wait complete. 2024-02-12T16:38:27.225 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:28.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:26.358508+0000 mgr.smithi104.ohwbhf (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:28.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:27.221287+0000 mon.smithi104 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.149:0/1321945668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:28.796 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:28.797 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:30.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:28.358904+0000 mgr.smithi104.ohwbhf (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:30.640 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.636+0000 7f60bbfff700 1 -- 172.21.15.149:0/3054569778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc0f62a0 msgr2=0x7f60bc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.636+0000 7f60bbfff700 1 --2- 172.21.15.149:0/3054569778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc0f62a0 0x7f60bc0f66c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f60ac009fb0 tx=0x7f60ac009b30).stop 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.636+0000 7f60bbfff700 1 -- 172.21.15.149:0/3054569778 shutdown_connections 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.636+0000 7f60bbfff700 1 --2- 172.21.15.149:0/3054569778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc0f62a0 0x7f60bc0f66c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.636+0000 7f60bbfff700 1 -- 172.21.15.149:0/3054569778 >> 172.21.15.149:0/3054569778 conn(0x7f60bc0f1aa0 msgr2=0x7f60bc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.637+0000 7f60bbfff700 1 -- 172.21.15.149:0/3054569778 shutdown_connections 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.637+0000 7f60bbfff700 1 -- 172.21.15.149:0/3054569778 wait complete. 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.637+0000 7f60bbfff700 1 Processor -- start 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.637+0000 7f60bbfff700 1 -- start start 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.637+0000 7f60bbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc0f62a0 0x7f60bc1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.637+0000 7f60bbfff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc106ac0 con 0x7f60bc0f62a0 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.638+0000 7f60baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc0f62a0 0x7f60bc1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:30.641 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.638+0000 7f60baffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc0f62a0 0x7f60bc1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:48106/0 (socket says 172.21.15.149:48106) 2024-02-12T16:38:30.642 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.638+0000 7f60baffd700 1 -- 172.21.15.149:0/489751992 learned_addr learned my addr 172.21.15.149:0/489751992 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:30.642 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.638+0000 7f60baffd700 1 -- 172.21.15.149:0/489751992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac005040 con 0x7f60bc0f62a0 2024-02-12T16:38:30.642 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.638+0000 7f60baffd700 1 --2- 172.21.15.149:0/489751992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc0f62a0 0x7f60bc1065d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f60ac0089f0 tx=0x7f60ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:30.642 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.638+0000 7f60a37fe700 1 -- 172.21.15.149:0/489751992 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac009320 con 0x7f60bc0f62a0 2024-02-12T16:38:30.642 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.638+0000 7f60bbfff700 1 -- 172.21.15.149:0/489751992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60bc106cc0 con 0x7f60bc0f62a0 2024-02-12T16:38:30.642 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.638+0000 7f60a37fe700 1 -- 172.21.15.149:0/489751992 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60ac013380 con 0x7f60bc0f62a0 2024-02-12T16:38:30.642 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.638+0000 7f60a37fe700 1 -- 172.21.15.149:0/489751992 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac014e20 con 0x7f60bc0f62a0 2024-02-12T16:38:30.642 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.638+0000 7f60bbfff700 1 -- 172.21.15.149:0/489751992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc107150 con 0x7f60bc0f62a0 2024-02-12T16:38:30.643 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.639+0000 7f60a37fe700 1 -- 172.21.15.149:0/489751992 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60ac0095f0 con 0x7f60bc0f62a0 2024-02-12T16:38:30.643 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.639+0000 7f60bbfff700 1 -- 172.21.15.149:0/489751992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60bc045ad0 con 0x7f60bc0f62a0 2024-02-12T16:38:30.643 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.639+0000 7f60a37fe700 1 --2- 172.21.15.149:0/489751992 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f60a402ff10 0x7f60a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:30.643 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.639+0000 7f60a37fe700 1 -- 172.21.15.149:0/489751992 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60ac040f50 con 0x7f60bc0f62a0 2024-02-12T16:38:30.643 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.639+0000 7f60ba7fc700 1 --2- 172.21.15.149:0/489751992 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f60a402ff10 0x7f60a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:30.643 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.640+0000 7f60ba7fc700 1 --2- 172.21.15.149:0/489751992 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f60a402ff10 0x7f60a40323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f60b00096a0 tx=0x7f60b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:30.646 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.645+0000 7f60a37fe700 1 -- 172.21.15.149:0/489751992 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60ac014420 con 0x7f60bc0f62a0 2024-02-12T16:38:30.851 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.849+0000 7f60bbfff700 1 -- 172.21.15.149:0/489751992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60bc108000 con 0x7f60bc0f62a0 2024-02-12T16:38:30.852 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.850+0000 7f60a37fe700 1 -- 172.21.15.149:0/489751992 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f60ac01d980 con 0x7f60bc0f62a0 2024-02-12T16:38:30.853 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:30.853 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:30.854 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.852+0000 7f60bbfff700 1 -- 172.21.15.149:0/489751992 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f60a402ff10 msgr2=0x7f60a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:30.854 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.852+0000 7f60bbfff700 1 --2- 172.21.15.149:0/489751992 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f60a402ff10 0x7f60a40323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f60b00096a0 tx=0x7f60b0006b40).stop 2024-02-12T16:38:30.854 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.852+0000 7f60bbfff700 1 -- 172.21.15.149:0/489751992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc0f62a0 msgr2=0x7f60bc1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:30.854 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.852+0000 7f60bbfff700 1 --2- 172.21.15.149:0/489751992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc0f62a0 0x7f60bc1065d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f60ac0089f0 tx=0x7f60ac00b040).stop 2024-02-12T16:38:30.854 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.853+0000 7f60bbfff700 1 -- 172.21.15.149:0/489751992 shutdown_connections 2024-02-12T16:38:30.855 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.853+0000 7f60bbfff700 1 --2- 172.21.15.149:0/489751992 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f60a402ff10 0x7f60a40323d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:30.855 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.853+0000 7f60bbfff700 1 --2- 172.21.15.149:0/489751992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60bc0f62a0 0x7f60bc1065d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:30.855 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.853+0000 7f60bbfff700 1 -- 172.21.15.149:0/489751992 >> 172.21.15.149:0/489751992 conn(0x7f60bc0f1aa0 msgr2=0x7f60bc0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:30.855 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.853+0000 7f60bbfff700 1 -- 172.21.15.149:0/489751992 shutdown_connections 2024-02-12T16:38:30.855 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:30.853+0000 7f60bbfff700 1 -- 172.21.15.149:0/489751992 wait complete. 2024-02-12T16:38:30.855 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:31.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:31 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:30.851098+0000 mon.smithi104 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.149:0/489751992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:32.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:30.359196+0000 mgr.smithi104.ohwbhf (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:32.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:32.556 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:34.468 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.462+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1245540876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d40f51a0 msgr2=0x7fc9d40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:34.468 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.462+0000 7fc9d98f1700 1 --2- 172.21.15.149:0/1245540876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d40f51a0 0x7fc9d40f55c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fc9c8009fb0 tx=0x7fc9c8009b30).stop 2024-02-12T16:38:34.468 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.462+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1245540876 shutdown_connections 2024-02-12T16:38:34.468 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.462+0000 7fc9d98f1700 1 --2- 172.21.15.149:0/1245540876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d40f51a0 0x7fc9d40f55c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:34.468 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.462+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1245540876 >> 172.21.15.149:0/1245540876 conn(0x7fc9d40f09a0 msgr2=0x7fc9d40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:34.468 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.462+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1245540876 shutdown_connections 2024-02-12T16:38:34.468 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.462+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1245540876 wait complete. 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.463+0000 7fc9d98f1700 1 Processor -- start 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.463+0000 7fc9d98f1700 1 -- start start 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.463+0000 7fc9d98f1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d4060200 0x7fc9d4109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.463+0000 7fc9d98f1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d4109900 con 0x7fc9d4060200 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.463+0000 7fc9d88ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d4060200 0x7fc9d4109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.463+0000 7fc9d88ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d4060200 0x7fc9d4109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:38574/0 (socket says 172.21.15.149:38574) 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.463+0000 7fc9d88ef700 1 -- 172.21.15.149:0/1426958089 learned_addr learned my addr 172.21.15.149:0/1426958089 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.463+0000 7fc9d88ef700 1 -- 172.21.15.149:0/1426958089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c8005040 con 0x7fc9d4060200 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.464+0000 7fc9d88ef700 1 --2- 172.21.15.149:0/1426958089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d4060200 0x7fc9d4109410 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fc9c8011000 tx=0x7fc9c8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.464+0000 7fc9d17fa700 1 -- 172.21.15.149:0/1426958089 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9c8013380 con 0x7fc9d4060200 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.464+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1426958089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d4109a40 con 0x7fc9d4060200 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.464+0000 7fc9d17fa700 1 -- 172.21.15.149:0/1426958089 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9c8013790 con 0x7fc9d4060200 2024-02-12T16:38:34.469 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.464+0000 7fc9d17fa700 1 -- 172.21.15.149:0/1426958089 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9c801ee70 con 0x7fc9d4060200 2024-02-12T16:38:34.470 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.464+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1426958089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d4109e70 con 0x7fc9d4060200 2024-02-12T16:38:34.470 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.465+0000 7fc9d17fa700 1 -- 172.21.15.149:0/1426958089 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9c8025060 con 0x7fc9d4060200 2024-02-12T16:38:34.470 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.465+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1426958089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d4045ad0 con 0x7fc9d4060200 2024-02-12T16:38:34.470 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.465+0000 7fc9d17fa700 1 --2- 172.21.15.149:0/1426958089 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc9c402ff60 0x7fc9c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:34.470 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.465+0000 7fc9d17fa700 1 -- 172.21.15.149:0/1426958089 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9c8040ea0 con 0x7fc9d4060200 2024-02-12T16:38:34.470 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.465+0000 7fc9d3fff700 1 --2- 172.21.15.149:0/1426958089 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc9c402ff60 0x7fc9c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:34.470 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.465+0000 7fc9d3fff700 1 --2- 172.21.15.149:0/1426958089 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc9c402ff60 0x7fc9c4032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fc9c00096a0 tx=0x7fc9c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:34.470 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.469+0000 7fc9d17fa700 1 -- 172.21.15.149:0/1426958089 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9c801c6c0 con 0x7fc9d4060200 2024-02-12T16:38:34.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:32.359593+0000 mgr.smithi104.ohwbhf (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:34.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.674+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1426958089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9d40fad30 con 0x7fc9d4060200 2024-02-12T16:38:34.676 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.675+0000 7fc9d17fa700 1 -- 172.21.15.149:0/1426958089 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc9c802d2d0 con 0x7fc9d4060200 2024-02-12T16:38:34.676 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:34.676 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:34.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.676+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1426958089 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc9c402ff60 msgr2=0x7fc9c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:34.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.676+0000 7fc9d98f1700 1 --2- 172.21.15.149:0/1426958089 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc9c402ff60 0x7fc9c4032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fc9c00096a0 tx=0x7fc9c0006b40).stop 2024-02-12T16:38:34.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.676+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1426958089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d4060200 msgr2=0x7fc9d4109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:34.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.676+0000 7fc9d98f1700 1 --2- 172.21.15.149:0/1426958089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d4060200 0x7fc9d4109410 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fc9c8011000 tx=0x7fc9c8009b30).stop 2024-02-12T16:38:34.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.676+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1426958089 shutdown_connections 2024-02-12T16:38:34.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.676+0000 7fc9d98f1700 1 --2- 172.21.15.149:0/1426958089 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc9c402ff60 0x7fc9c4032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:34.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.676+0000 7fc9d98f1700 1 --2- 172.21.15.149:0/1426958089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9d4060200 0x7fc9d4109410 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:34.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.676+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1426958089 >> 172.21.15.149:0/1426958089 conn(0x7fc9d40f09a0 msgr2=0x7fc9d40f14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:34.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.676+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1426958089 shutdown_connections 2024-02-12T16:38:34.678 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:34.676+0000 7fc9d98f1700 1 -- 172.21.15.149:0/1426958089 wait complete. 2024-02-12T16:38:34.679 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:35.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:35 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:34.675091+0000 mon.smithi104 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.149:0/1426958089' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:36.315 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:36.315 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:36.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:34.360028+0000 mgr.smithi104.ohwbhf (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:38.153 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.149+0000 7f9192d34700 1 -- 172.21.15.149:0/1444178877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f918c0f5410 msgr2=0x7f918c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:38.153 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.149+0000 7f9192d34700 1 --2- 172.21.15.149:0/1444178877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f918c0f5410 0x7f918c0f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f9180009fb0 tx=0x7f9180009b30).stop 2024-02-12T16:38:38.153 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.149+0000 7f9192d34700 1 -- 172.21.15.149:0/1444178877 shutdown_connections 2024-02-12T16:38:38.153 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.149+0000 7f9192d34700 1 --2- 172.21.15.149:0/1444178877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f918c0f5410 0x7f918c0f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:38.153 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.149+0000 7f9192d34700 1 -- 172.21.15.149:0/1444178877 >> 172.21.15.149:0/1444178877 conn(0x7f918c0f0b70 msgr2=0x7f918c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.149+0000 7f9192d34700 1 -- 172.21.15.149:0/1444178877 shutdown_connections 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.149+0000 7f9192d34700 1 -- 172.21.15.149:0/1444178877 wait complete. 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.149+0000 7f9192d34700 1 Processor -- start 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.150+0000 7f9192d34700 1 -- start start 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.150+0000 7f9192d34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f918c0f5410 0x7f918c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.150+0000 7f9192d34700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c105bd0 con 0x7f918c0f5410 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.150+0000 7f9190ad0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f918c0f5410 0x7f918c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.150+0000 7f9190ad0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f918c0f5410 0x7f918c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:38598/0 (socket says 172.21.15.149:38598) 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.150+0000 7f9190ad0700 1 -- 172.21.15.149:0/754169892 learned_addr learned my addr 172.21.15.149:0/754169892 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.150+0000 7f9190ad0700 1 -- 172.21.15.149:0/754169892 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9180005040 con 0x7f918c0f5410 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.151+0000 7f9190ad0700 1 --2- 172.21.15.149:0/754169892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f918c0f5410 0x7f918c1056e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f9180005010 tx=0x7f918000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.151+0000 7f91897fa700 1 -- 172.21.15.149:0/754169892 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9180009320 con 0x7f918c0f5410 2024-02-12T16:38:38.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.151+0000 7f9192d34700 1 -- 172.21.15.149:0/754169892 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918c105dd0 con 0x7f918c0f5410 2024-02-12T16:38:38.155 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.151+0000 7f91897fa700 1 -- 172.21.15.149:0/754169892 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9180013380 con 0x7f918c0f5410 2024-02-12T16:38:38.155 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.151+0000 7f91897fa700 1 -- 172.21.15.149:0/754169892 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9180014e50 con 0x7f918c0f5410 2024-02-12T16:38:38.155 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.151+0000 7f9192d34700 1 -- 172.21.15.149:0/754169892 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c106260 con 0x7f918c0f5410 2024-02-12T16:38:38.155 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.152+0000 7f91897fa700 1 -- 172.21.15.149:0/754169892 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91800095f0 con 0x7f918c0f5410 2024-02-12T16:38:38.156 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.152+0000 7f91897fa700 1 --2- 172.21.15.149:0/754169892 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f917c02ff10 0x7f917c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:38.156 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.152+0000 7f9192d34700 1 -- 172.21.15.149:0/754169892 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918c045ad0 con 0x7f918c0f5410 2024-02-12T16:38:38.156 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.152+0000 7f91897fa700 1 -- 172.21.15.149:0/754169892 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9180040e90 con 0x7f918c0f5410 2024-02-12T16:38:38.156 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.152+0000 7f918bfff700 1 --2- 172.21.15.149:0/754169892 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f917c02ff10 0x7f917c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:38.156 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.152+0000 7f918bfff700 1 --2- 172.21.15.149:0/754169892 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f917c02ff10 0x7f917c0323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f91780096a0 tx=0x7f9178006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:38.158 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.156+0000 7f91897fa700 1 -- 172.21.15.149:0/754169892 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9180014420 con 0x7f918c0f5410 2024-02-12T16:38:38.362 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.360+0000 7f9192d34700 1 -- 172.21.15.149:0/754169892 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f918c107110 con 0x7f918c0f5410 2024-02-12T16:38:38.363 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.361+0000 7f91897fa700 1 -- 172.21.15.149:0/754169892 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9180010230 con 0x7f918c0f5410 2024-02-12T16:38:38.363 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:38.364 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:38.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.363+0000 7f9192d34700 1 -- 172.21.15.149:0/754169892 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f917c02ff10 msgr2=0x7f917c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:38.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.363+0000 7f9192d34700 1 --2- 172.21.15.149:0/754169892 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f917c02ff10 0x7f917c0323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f91780096a0 tx=0x7f9178006b40).stop 2024-02-12T16:38:38.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.363+0000 7f9192d34700 1 -- 172.21.15.149:0/754169892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f918c0f5410 msgr2=0x7f918c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:38.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.363+0000 7f9192d34700 1 --2- 172.21.15.149:0/754169892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f918c0f5410 0x7f918c1056e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f9180005010 tx=0x7f918000b040).stop 2024-02-12T16:38:38.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.363+0000 7f9192d34700 1 -- 172.21.15.149:0/754169892 shutdown_connections 2024-02-12T16:38:38.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.363+0000 7f9192d34700 1 --2- 172.21.15.149:0/754169892 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f917c02ff10 0x7f917c0323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:38.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.363+0000 7f9192d34700 1 --2- 172.21.15.149:0/754169892 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f918c0f5410 0x7f918c1056e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:38.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.363+0000 7f9192d34700 1 -- 172.21.15.149:0/754169892 >> 172.21.15.149:0/754169892 conn(0x7f918c0f0b70 msgr2=0x7f918c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:38.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.363+0000 7f9192d34700 1 -- 172.21.15.149:0/754169892 shutdown_connections 2024-02-12T16:38:38.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:38.363+0000 7f9192d34700 1 -- 172.21.15.149:0/754169892 wait complete. 2024-02-12T16:38:38.366 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:38.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:36.360446+0000 mgr.smithi104.ohwbhf (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:39.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:39 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:38.362388+0000 mon.smithi104 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.149:0/754169892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:39.945 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:39.945 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:40.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:38.360762+0000 mgr.smithi104.ohwbhf (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:41.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.787+0000 7f2cbdbef700 1 -- 172.21.15.149:0/3377641966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb80f51a0 msgr2=0x7f2cb80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:41.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.787+0000 7f2cbdbef700 1 --2- 172.21.15.149:0/3377641966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb80f51a0 0x7f2cb80f55c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f2cac009fb0 tx=0x7f2cac009b30).stop 2024-02-12T16:38:41.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.787+0000 7f2cbdbef700 1 -- 172.21.15.149:0/3377641966 shutdown_connections 2024-02-12T16:38:41.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.787+0000 7f2cbdbef700 1 --2- 172.21.15.149:0/3377641966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb80f51a0 0x7f2cb80f55c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:41.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.787+0000 7f2cbdbef700 1 -- 172.21.15.149:0/3377641966 >> 172.21.15.149:0/3377641966 conn(0x7f2cb80f09a0 msgr2=0x7f2cb80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:41.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.787+0000 7f2cbdbef700 1 -- 172.21.15.149:0/3377641966 shutdown_connections 2024-02-12T16:38:41.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.787+0000 7f2cbdbef700 1 -- 172.21.15.149:0/3377641966 wait complete. 2024-02-12T16:38:41.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.787+0000 7f2cbdbef700 1 Processor -- start 2024-02-12T16:38:41.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.787+0000 7f2cbdbef700 1 -- start start 2024-02-12T16:38:41.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.787+0000 7f2cbdbef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb80f51a0 0x7f2cb81054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:41.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.787+0000 7f2cbdbef700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb81059c0 con 0x7f2cb80f51a0 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.788+0000 7f2cbcbed700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb80f51a0 0x7f2cb81054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.788+0000 7f2cbcbed700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb80f51a0 0x7f2cb81054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:38604/0 (socket says 172.21.15.149:38604) 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.788+0000 7f2cbcbed700 1 -- 172.21.15.149:0/908046811 learned_addr learned my addr 172.21.15.149:0/908046811 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.788+0000 7f2cbcbed700 1 -- 172.21.15.149:0/908046811 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cac005040 con 0x7f2cb80f51a0 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.788+0000 7f2cbcbed700 1 --2- 172.21.15.149:0/908046811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb80f51a0 0x7f2cb81054d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f2cac0089f0 tx=0x7f2cac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.788+0000 7f2cb57fa700 1 -- 172.21.15.149:0/908046811 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cac009320 con 0x7f2cb80f51a0 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.788+0000 7f2cbdbef700 1 -- 172.21.15.149:0/908046811 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb8105bc0 con 0x7f2cb80f51a0 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.788+0000 7f2cb57fa700 1 -- 172.21.15.149:0/908046811 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cac013380 con 0x7f2cb80f51a0 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.788+0000 7f2cbdbef700 1 -- 172.21.15.149:0/908046811 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb8106050 con 0x7f2cb80f51a0 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.788+0000 7f2cb57fa700 1 -- 172.21.15.149:0/908046811 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cac014e20 con 0x7f2cb80f51a0 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.789+0000 7f2cb57fa700 1 -- 172.21.15.149:0/908046811 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2cac0095f0 con 0x7f2cb80f51a0 2024-02-12T16:38:41.794 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.789+0000 7f2cbdbef700 1 -- 172.21.15.149:0/908046811 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb8045ad0 con 0x7f2cb80f51a0 2024-02-12T16:38:41.795 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.789+0000 7f2cb57fa700 1 --2- 172.21.15.149:0/908046811 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2ca8032120 0x7f2ca80345e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:41.795 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.789+0000 7f2cb57fa700 1 -- 172.21.15.149:0/908046811 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2cac040f60 con 0x7f2cb80f51a0 2024-02-12T16:38:41.795 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.790+0000 7f2cb7fff700 1 --2- 172.21.15.149:0/908046811 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2ca8032120 0x7f2ca80345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:41.795 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.790+0000 7f2cb7fff700 1 --2- 172.21.15.149:0/908046811 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2ca8032120 0x7f2ca80345e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f2ca40096a0 tx=0x7f2ca4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:41.796 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.794+0000 7f2cb57fa700 1 -- 172.21.15.149:0/908046811 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2cac014420 con 0x7f2cb80f51a0 2024-02-12T16:38:42.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.999+0000 7f2cbdbef700 1 -- 172.21.15.149:0/908046811 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2cb8106ea0 con 0x7f2cb80f51a0 2024-02-12T16:38:42.001 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:41.999+0000 7f2cb57fa700 1 -- 172.21.15.149:0/908046811 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2cac010230 con 0x7f2cb80f51a0 2024-02-12T16:38:42.002 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:42.002 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:42.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:42.001+0000 7f2cbdbef700 1 -- 172.21.15.149:0/908046811 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2ca8032120 msgr2=0x7f2ca80345e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:42.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:42.001+0000 7f2cbdbef700 1 --2- 172.21.15.149:0/908046811 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2ca8032120 0x7f2ca80345e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f2ca40096a0 tx=0x7f2ca4006b40).stop 2024-02-12T16:38:42.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:42.001+0000 7f2cbdbef700 1 -- 172.21.15.149:0/908046811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb80f51a0 msgr2=0x7f2cb81054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:42.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:42.001+0000 7f2cbdbef700 1 --2- 172.21.15.149:0/908046811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb80f51a0 0x7f2cb81054d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f2cac0089f0 tx=0x7f2cac00b040).stop 2024-02-12T16:38:42.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:42.001+0000 7f2cbdbef700 1 -- 172.21.15.149:0/908046811 shutdown_connections 2024-02-12T16:38:42.003 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:42.001+0000 7f2cbdbef700 1 --2- 172.21.15.149:0/908046811 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2ca8032120 0x7f2ca80345e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:42.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:42.001+0000 7f2cbdbef700 1 --2- 172.21.15.149:0/908046811 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cb80f51a0 0x7f2cb81054d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:42.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:42.001+0000 7f2cbdbef700 1 -- 172.21.15.149:0/908046811 >> 172.21.15.149:0/908046811 conn(0x7f2cb80f09a0 msgr2=0x7f2cb80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:42.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:42.001+0000 7f2cbdbef700 1 -- 172.21.15.149:0/908046811 shutdown_connections 2024-02-12T16:38:42.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:42.001+0000 7f2cbdbef700 1 -- 172.21.15.149:0/908046811 wait complete. 2024-02-12T16:38:42.004 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:42.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:40.361134+0000 mgr.smithi104.ohwbhf (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:42.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:42.000785+0000 mon.smithi104 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.149:0/908046811' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:43.661 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:43.661 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:44.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:42.361471+0000 mgr.smithi104.ohwbhf (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:45.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.487+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3426743711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 msgr2=0x7fb8cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:45.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.487+0000 7fb8d1f44700 1 --2- 172.21.15.149:0/3426743711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc0f6930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fb8bc009fb0 tx=0x7fb8bc009b30).stop 2024-02-12T16:38:45.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.487+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3426743711 shutdown_connections 2024-02-12T16:38:45.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.487+0000 7fb8d1f44700 1 --2- 172.21.15.149:0/3426743711 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc0f6930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:45.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.487+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3426743711 >> 172.21.15.149:0/3426743711 conn(0x7fb8cc0f1c70 msgr2=0x7fb8cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:45.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.487+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3426743711 shutdown_connections 2024-02-12T16:38:45.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.487+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3426743711 wait complete. 2024-02-12T16:38:45.491 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.488+0000 7fb8d1f44700 1 Processor -- start 2024-02-12T16:38:45.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.488+0000 7fb8d1f44700 1 -- start start 2024-02-12T16:38:45.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.488+0000 7fb8d1f44700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:45.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.488+0000 7fb8d1f44700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc106cd0 con 0x7fb8cc0f6510 2024-02-12T16:38:45.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.489+0000 7fb8cb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:45.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.489+0000 7fb8cb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:54204/0 (socket says 172.21.15.149:54204) 2024-02-12T16:38:45.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.489+0000 7fb8cb7fe700 1 -- 172.21.15.149:0/3356823742 learned_addr learned my addr 172.21.15.149:0/3356823742 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:45.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.489+0000 7fb8cb7fe700 1 -- 172.21.15.149:0/3356823742 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc005040 con 0x7fb8cc0f6510 2024-02-12T16:38:45.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.489+0000 7fb8cb7fe700 1 --2- 172.21.15.149:0/3356823742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fb8bc005010 tx=0x7fb8bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:45.493 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.489+0000 7fb8b3fff700 1 -- 172.21.15.149:0/3356823742 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8bc009320 con 0x7fb8cc0f6510 2024-02-12T16:38:45.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.489+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3356823742 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8cc106ed0 con 0x7fb8cc0f6510 2024-02-12T16:38:45.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.489+0000 7fb8b3fff700 1 -- 172.21.15.149:0/3356823742 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8bc013380 con 0x7fb8cc0f6510 2024-02-12T16:38:45.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.489+0000 7fb8b3fff700 1 -- 172.21.15.149:0/3356823742 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8bc014e50 con 0x7fb8cc0f6510 2024-02-12T16:38:45.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.489+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3356823742 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8cc107360 con 0x7fb8cc0f6510 2024-02-12T16:38:45.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.490+0000 7fb8b3fff700 1 -- 172.21.15.149:0/3356823742 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8bc0095f0 con 0x7fb8cc0f6510 2024-02-12T16:38:45.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.490+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3356823742 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8cc045ad0 con 0x7fb8cc0f6510 2024-02-12T16:38:45.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.490+0000 7fb8b3fff700 1 --2- 172.21.15.149:0/3356823742 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 0x7fb8b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:45.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.490+0000 7fb8b3fff700 1 -- 172.21.15.149:0/3356823742 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8bc040e90 con 0x7fb8cc0f6510 2024-02-12T16:38:45.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.490+0000 7fb8caffd700 1 --2- 172.21.15.149:0/3356823742 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 0x7fb8b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:45.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.491+0000 7fb8caffd700 1 --2- 172.21.15.149:0/3356823742 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 0x7fb8b40323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fb8c00096a0 tx=0x7fb8c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:45.497 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.495+0000 7fb8b3fff700 1 -- 172.21.15.149:0/3356823742 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8bc014420 con 0x7fb8cc0f6510 2024-02-12T16:38:45.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.697+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3356823742 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8cc108210 con 0x7fb8cc0f6510 2024-02-12T16:38:45.700 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.697+0000 7fb8b3fff700 1 -- 172.21.15.149:0/3356823742 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb8bc010230 con 0x7fb8cc0f6510 2024-02-12T16:38:45.700 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:45.700 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:45.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.699+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3356823742 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 msgr2=0x7fb8b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:45.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.699+0000 7fb8d1f44700 1 --2- 172.21.15.149:0/3356823742 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 0x7fb8b40323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fb8c00096a0 tx=0x7fb8c0006b40).stop 2024-02-12T16:38:45.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.699+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3356823742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 msgr2=0x7fb8cc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:45.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.699+0000 7fb8d1f44700 1 --2- 172.21.15.149:0/3356823742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fb8bc005010 tx=0x7fb8bc00b040).stop 2024-02-12T16:38:45.702 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.699+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3356823742 shutdown_connections 2024-02-12T16:38:45.702 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.699+0000 7fb8d1f44700 1 --2- 172.21.15.149:0/3356823742 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb8b402ff10 0x7fb8b40323d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:45.702 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.699+0000 7fb8d1f44700 1 --2- 172.21.15.149:0/3356823742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8cc0f6510 0x7fb8cc1067e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:45.702 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.699+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3356823742 >> 172.21.15.149:0/3356823742 conn(0x7fb8cc0f1c70 msgr2=0x7fb8cc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:45.702 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.699+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3356823742 shutdown_connections 2024-02-12T16:38:45.702 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:45.699+0000 7fb8d1f44700 1 -- 172.21.15.149:0/3356823742 wait complete. 2024-02-12T16:38:45.703 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:46.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:44.361754+0000 mgr.smithi104.ohwbhf (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:46.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:45.698769+0000 mon.smithi104 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.149:0/3356823742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:47.252 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:47.253 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:48.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:46.362079+0000 mgr.smithi104.ohwbhf (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:49.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.148+0000 7f2c11177700 1 -- 172.21.15.149:0/2300775252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c0c0f5410 msgr2=0x7f2c0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:49.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.148+0000 7f2c11177700 1 --2- 172.21.15.149:0/2300775252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c0c0f5410 0x7f2c0c0f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f2c00009fb0 tx=0x7f2c00009b30).stop 2024-02-12T16:38:49.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.148+0000 7f2c11177700 1 -- 172.21.15.149:0/2300775252 shutdown_connections 2024-02-12T16:38:49.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.148+0000 7f2c11177700 1 --2- 172.21.15.149:0/2300775252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c0c0f5410 0x7f2c0c0f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:49.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.148+0000 7f2c11177700 1 -- 172.21.15.149:0/2300775252 >> 172.21.15.149:0/2300775252 conn(0x7f2c0c0f0b70 msgr2=0x7f2c0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:49.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.148+0000 7f2c11177700 1 -- 172.21.15.149:0/2300775252 shutdown_connections 2024-02-12T16:38:49.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.148+0000 7f2c11177700 1 -- 172.21.15.149:0/2300775252 wait complete. 2024-02-12T16:38:49.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.149+0000 7f2c11177700 1 Processor -- start 2024-02-12T16:38:49.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.149+0000 7f2c11177700 1 -- start start 2024-02-12T16:38:49.151 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.149+0000 7f2c11177700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c0c0f5410 0x7f2c0c109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:49.152 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.149+0000 7f2c11177700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c060220 con 0x7f2c0c0f5410 2024-02-12T16:38:49.153 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.149+0000 7f2c0ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c0c0f5410 0x7f2c0c109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:49.153 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.149+0000 7f2c0ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c0c0f5410 0x7f2c0c109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:54232/0 (socket says 172.21.15.149:54232) 2024-02-12T16:38:49.153 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.149+0000 7f2c0ad9d700 1 -- 172.21.15.149:0/3266581654 learned_addr learned my addr 172.21.15.149:0/3266581654 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.149+0000 7f2c0ad9d700 1 -- 172.21.15.149:0/3266581654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c00005040 con 0x7f2c0c0f5410 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.150+0000 7f2c0ad9d700 1 --2- 172.21.15.149:0/3266581654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c0c0f5410 0x7f2c0c109540 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f2c00005010 tx=0x7f2c0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.150+0000 7f2bf37fe700 1 -- 172.21.15.149:0/3266581654 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c00009320 con 0x7f2c0c0f5410 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.150+0000 7f2c11177700 1 -- 172.21.15.149:0/3266581654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0c060420 con 0x7f2c0c0f5410 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.150+0000 7f2bf37fe700 1 -- 172.21.15.149:0/3266581654 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c00013380 con 0x7f2c0c0f5410 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.150+0000 7f2bf37fe700 1 -- 172.21.15.149:0/3266581654 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c00014e50 con 0x7f2c0c0f5410 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.150+0000 7f2c11177700 1 -- 172.21.15.149:0/3266581654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c0c109bf0 con 0x7f2c0c0f5410 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.151+0000 7f2bf37fe700 1 -- 172.21.15.149:0/3266581654 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c000095f0 con 0x7f2c0c0f5410 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.151+0000 7f2c11177700 1 -- 172.21.15.149:0/3266581654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0c059100 con 0x7f2c0c0f5410 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.151+0000 7f2bf37fe700 1 --2- 172.21.15.149:0/3266581654 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2bf402ff10 0x7f2bf40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.151+0000 7f2bf37fe700 1 -- 172.21.15.149:0/3266581654 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c00040f60 con 0x7f2c0c0f5410 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.151+0000 7f2c0a59c700 1 --2- 172.21.15.149:0/3266581654 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2bf402ff10 0x7f2bf40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:49.154 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.151+0000 7f2c0a59c700 1 --2- 172.21.15.149:0/3266581654 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2bf402ff10 0x7f2bf40323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f2bfc0096a0 tx=0x7f2bfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:49.157 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.155+0000 7f2bf37fe700 1 -- 172.21.15.149:0/3266581654 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c00014420 con 0x7f2c0c0f5410 2024-02-12T16:38:49.363 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.361+0000 7f2c11177700 1 -- 172.21.15.149:0/3266581654 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c0c045ad0 con 0x7f2c0c0f5410 2024-02-12T16:38:49.364 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.361+0000 7f2bf37fe700 1 -- 172.21.15.149:0/3266581654 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2c00014420 con 0x7f2c0c0f5410 2024-02-12T16:38:49.364 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:49.365 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:49.365 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.363+0000 7f2c11177700 1 -- 172.21.15.149:0/3266581654 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2bf402ff10 msgr2=0x7f2bf40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:49.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.363+0000 7f2c11177700 1 --2- 172.21.15.149:0/3266581654 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2bf402ff10 0x7f2bf40323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f2bfc0096a0 tx=0x7f2bfc006b40).stop 2024-02-12T16:38:49.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.363+0000 7f2c11177700 1 -- 172.21.15.149:0/3266581654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c0c0f5410 msgr2=0x7f2c0c109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:49.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.363+0000 7f2c11177700 1 --2- 172.21.15.149:0/3266581654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c0c0f5410 0x7f2c0c109540 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f2c00005010 tx=0x7f2c0000b040).stop 2024-02-12T16:38:49.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.363+0000 7f2c11177700 1 -- 172.21.15.149:0/3266581654 shutdown_connections 2024-02-12T16:38:49.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.363+0000 7f2c11177700 1 --2- 172.21.15.149:0/3266581654 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f2bf402ff10 0x7f2bf40323d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:49.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.363+0000 7f2c11177700 1 --2- 172.21.15.149:0/3266581654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2c0c0f5410 0x7f2c0c109540 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:49.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.363+0000 7f2c11177700 1 -- 172.21.15.149:0/3266581654 >> 172.21.15.149:0/3266581654 conn(0x7f2c0c0f0b70 msgr2=0x7f2c0c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:49.366 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.363+0000 7f2c11177700 1 -- 172.21.15.149:0/3266581654 shutdown_connections 2024-02-12T16:38:49.367 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:49.363+0000 7f2c11177700 1 -- 172.21.15.149:0/3266581654 wait complete. 2024-02-12T16:38:49.367 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:50.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:48.362392+0000 mgr.smithi104.ohwbhf (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:50.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:49.362824+0000 mon.smithi104 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.149:0/3266581654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:50.927 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:50.927 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:52.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:50.362714+0000 mgr.smithi104.ohwbhf (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:52.839 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.835+0000 7fed0b981700 1 -- 172.21.15.149:0/2900238619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed040f5410 msgr2=0x7fed040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:52.839 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.835+0000 7fed0b981700 1 --2- 172.21.15.149:0/2900238619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed040f5410 0x7fed040f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fecf4009fb0 tx=0x7fecf4009b30).stop 2024-02-12T16:38:52.839 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.835+0000 7fed0b981700 1 -- 172.21.15.149:0/2900238619 shutdown_connections 2024-02-12T16:38:52.839 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.835+0000 7fed0b981700 1 --2- 172.21.15.149:0/2900238619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed040f5410 0x7fed040f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:52.839 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.835+0000 7fed0b981700 1 -- 172.21.15.149:0/2900238619 >> 172.21.15.149:0/2900238619 conn(0x7fed040f0b70 msgr2=0x7fed040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:52.839 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.835+0000 7fed0b981700 1 -- 172.21.15.149:0/2900238619 shutdown_connections 2024-02-12T16:38:52.839 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.835+0000 7fed0b981700 1 -- 172.21.15.149:0/2900238619 wait complete. 2024-02-12T16:38:52.839 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.836+0000 7fed0b981700 1 Processor -- start 2024-02-12T16:38:52.839 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.836+0000 7fed0b981700 1 -- start start 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.836+0000 7fed0b981700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed040f5410 0x7fed041067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.836+0000 7fed0b981700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed04106cd0 con 0x7fed040f5410 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.836+0000 7fed0971d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed040f5410 0x7fed041067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.836+0000 7fed0971d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed040f5410 0x7fed041067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:54250/0 (socket says 172.21.15.149:54250) 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.836+0000 7fed0971d700 1 -- 172.21.15.149:0/511217098 learned_addr learned my addr 172.21.15.149:0/511217098 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.836+0000 7fed0971d700 1 -- 172.21.15.149:0/511217098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf4005040 con 0x7fed040f5410 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.836+0000 7fed0971d700 1 --2- 172.21.15.149:0/511217098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed040f5410 0x7fed041067e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fecf4005010 tx=0x7fecf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.837+0000 7fecf9ffb700 1 -- 172.21.15.149:0/511217098 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecf4009320 con 0x7fed040f5410 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.837+0000 7fed0b981700 1 -- 172.21.15.149:0/511217098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed04106ed0 con 0x7fed040f5410 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.837+0000 7fecf9ffb700 1 -- 172.21.15.149:0/511217098 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fecf4013380 con 0x7fed040f5410 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.837+0000 7fecf9ffb700 1 -- 172.21.15.149:0/511217098 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecf4014e50 con 0x7fed040f5410 2024-02-12T16:38:52.840 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.837+0000 7fed0b981700 1 -- 172.21.15.149:0/511217098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed04107360 con 0x7fed040f5410 2024-02-12T16:38:52.841 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.838+0000 7fecf9ffb700 1 -- 172.21.15.149:0/511217098 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fecf40095f0 con 0x7fed040f5410 2024-02-12T16:38:52.841 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.838+0000 7fed0b981700 1 -- 172.21.15.149:0/511217098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed04045ad0 con 0x7fed040f5410 2024-02-12T16:38:52.842 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.838+0000 7fecf9ffb700 1 --2- 172.21.15.149:0/511217098 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fecf002ff10 0x7fecf00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:52.842 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.838+0000 7fecf9ffb700 1 -- 172.21.15.149:0/511217098 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecf4040f10 con 0x7fed040f5410 2024-02-12T16:38:52.842 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.838+0000 7fed08f1c700 1 --2- 172.21.15.149:0/511217098 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fecf002ff10 0x7fecf00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:52.842 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.838+0000 7fed08f1c700 1 --2- 172.21.15.149:0/511217098 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fecf002ff10 0x7fecf00323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fed000096a0 tx=0x7fed00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:52.844 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:52.842+0000 7fecf9ffb700 1 -- 172.21.15.149:0/511217098 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecf4014420 con 0x7fed040f5410 2024-02-12T16:38:53.048 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.046+0000 7fed0b981700 1 -- 172.21.15.149:0/511217098 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed040fafa0 con 0x7fed040f5410 2024-02-12T16:38:53.049 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.047+0000 7fecf9ffb700 1 -- 172.21.15.149:0/511217098 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fecf4010230 con 0x7fed040f5410 2024-02-12T16:38:53.049 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:53.049 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:53.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.048+0000 7fed0b981700 1 -- 172.21.15.149:0/511217098 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fecf002ff10 msgr2=0x7fecf00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:53.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.048+0000 7fed0b981700 1 --2- 172.21.15.149:0/511217098 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fecf002ff10 0x7fecf00323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fed000096a0 tx=0x7fed00006b40).stop 2024-02-12T16:38:53.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.048+0000 7fed0b981700 1 -- 172.21.15.149:0/511217098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed040f5410 msgr2=0x7fed041067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:53.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.048+0000 7fed0b981700 1 --2- 172.21.15.149:0/511217098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed040f5410 0x7fed041067e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fecf4005010 tx=0x7fecf400b040).stop 2024-02-12T16:38:53.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.048+0000 7fed0b981700 1 -- 172.21.15.149:0/511217098 shutdown_connections 2024-02-12T16:38:53.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.048+0000 7fed0b981700 1 --2- 172.21.15.149:0/511217098 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fecf002ff10 0x7fecf00323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:53.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.048+0000 7fed0b981700 1 --2- 172.21.15.149:0/511217098 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed040f5410 0x7fed041067e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:53.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.048+0000 7fed0b981700 1 -- 172.21.15.149:0/511217098 >> 172.21.15.149:0/511217098 conn(0x7fed040f0b70 msgr2=0x7fed0405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:53.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.049+0000 7fed0b981700 1 -- 172.21.15.149:0/511217098 shutdown_connections 2024-02-12T16:38:53.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:53.049+0000 7fed0b981700 1 -- 172.21.15.149:0/511217098 wait complete. 2024-02-12T16:38:53.052 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:53.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:53 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:53.048307+0000 mon.smithi104 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.149:0/511217098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:54.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:52.362989+0000 mgr.smithi104.ohwbhf (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:54.682 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:54.682 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:56.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.480+0000 7fd5c45e6700 1 -- 172.21.15.149:0/378508850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5bc0f5410 msgr2=0x7fd5bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:56.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.480+0000 7fd5c45e6700 1 --2- 172.21.15.149:0/378508850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5bc0f5410 0x7fd5bc0f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fd5ac009fb0 tx=0x7fd5ac009b30).stop 2024-02-12T16:38:56.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.480+0000 7fd5c45e6700 1 -- 172.21.15.149:0/378508850 shutdown_connections 2024-02-12T16:38:56.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.480+0000 7fd5c45e6700 1 --2- 172.21.15.149:0/378508850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5bc0f5410 0x7fd5bc0f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:56.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.480+0000 7fd5c45e6700 1 -- 172.21.15.149:0/378508850 >> 172.21.15.149:0/378508850 conn(0x7fd5bc0f0b70 msgr2=0x7fd5bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.480+0000 7fd5c45e6700 1 -- 172.21.15.149:0/378508850 shutdown_connections 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.480+0000 7fd5c45e6700 1 -- 172.21.15.149:0/378508850 wait complete. 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.480+0000 7fd5c45e6700 1 Processor -- start 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.481+0000 7fd5c45e6700 1 -- start start 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.481+0000 7fd5c45e6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5bc0f5410 0x7fd5bc1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.481+0000 7fd5c45e6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc106cc0 con 0x7fd5bc0f5410 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.481+0000 7fd5c2382700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5bc0f5410 0x7fd5bc1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.481+0000 7fd5c2382700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5bc0f5410 0x7fd5bc1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:59150/0 (socket says 172.21.15.149:59150) 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.481+0000 7fd5c2382700 1 -- 172.21.15.149:0/4012112 learned_addr learned my addr 172.21.15.149:0/4012112 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.481+0000 7fd5c2382700 1 -- 172.21.15.149:0/4012112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ac005040 con 0x7fd5bc0f5410 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.481+0000 7fd5c2382700 1 --2- 172.21.15.149:0/4012112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5bc0f5410 0x7fd5bc1067d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd5ac005010 tx=0x7fd5ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.482+0000 7fd5b2ffd700 1 -- 172.21.15.149:0/4012112 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5ac009320 con 0x7fd5bc0f5410 2024-02-12T16:38:56.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.482+0000 7fd5c45e6700 1 -- 172.21.15.149:0/4012112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5bc106ec0 con 0x7fd5bc0f5410 2024-02-12T16:38:56.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.482+0000 7fd5b2ffd700 1 -- 172.21.15.149:0/4012112 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5ac013380 con 0x7fd5bc0f5410 2024-02-12T16:38:56.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.482+0000 7fd5b2ffd700 1 -- 172.21.15.149:0/4012112 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5ac014e50 con 0x7fd5bc0f5410 2024-02-12T16:38:56.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.482+0000 7fd5c45e6700 1 -- 172.21.15.149:0/4012112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5bc107350 con 0x7fd5bc0f5410 2024-02-12T16:38:56.486 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.482+0000 7fd5c45e6700 1 -- 172.21.15.149:0/4012112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5bc045ad0 con 0x7fd5bc0f5410 2024-02-12T16:38:56.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.482+0000 7fd5b2ffd700 1 -- 172.21.15.149:0/4012112 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5ac014420 con 0x7fd5bc0f5410 2024-02-12T16:38:56.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.483+0000 7fd5b2ffd700 1 --2- 172.21.15.149:0/4012112 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd5a802ff10 0x7fd5a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:38:56.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.483+0000 7fd5b2ffd700 1 -- 172.21.15.149:0/4012112 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5ac040e60 con 0x7fd5bc0f5410 2024-02-12T16:38:56.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.483+0000 7fd5c1b81700 1 --2- 172.21.15.149:0/4012112 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd5a802ff10 0x7fd5a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:38:56.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.483+0000 7fd5c1b81700 1 --2- 172.21.15.149:0/4012112 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd5a802ff10 0x7fd5a80323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd5b80096a0 tx=0x7fd5b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:38:56.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.487+0000 7fd5b2ffd700 1 -- 172.21.15.149:0/4012112 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5ac0095f0 con 0x7fd5bc0f5410 2024-02-12T16:38:56.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:54.363214+0000 mgr.smithi104.ohwbhf (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:38:56.695 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.693+0000 7fd5c45e6700 1 -- 172.21.15.149:0/4012112 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5bc108200 con 0x7fd5bc0f5410 2024-02-12T16:38:56.696 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.694+0000 7fd5b2ffd700 1 -- 172.21.15.149:0/4012112 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd5ac01db70 con 0x7fd5bc0f5410 2024-02-12T16:38:56.696 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:38:56.697 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:38:56.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.696+0000 7fd5c45e6700 1 -- 172.21.15.149:0/4012112 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd5a802ff10 msgr2=0x7fd5a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:56.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.696+0000 7fd5c45e6700 1 --2- 172.21.15.149:0/4012112 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd5a802ff10 0x7fd5a80323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd5b80096a0 tx=0x7fd5b8006b40).stop 2024-02-12T16:38:56.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.696+0000 7fd5c45e6700 1 -- 172.21.15.149:0/4012112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5bc0f5410 msgr2=0x7fd5bc1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:38:56.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.696+0000 7fd5c45e6700 1 --2- 172.21.15.149:0/4012112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5bc0f5410 0x7fd5bc1067d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd5ac005010 tx=0x7fd5ac00b040).stop 2024-02-12T16:38:56.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.696+0000 7fd5c45e6700 1 -- 172.21.15.149:0/4012112 shutdown_connections 2024-02-12T16:38:56.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.696+0000 7fd5c45e6700 1 --2- 172.21.15.149:0/4012112 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd5a802ff10 0x7fd5a80323d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:56.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.696+0000 7fd5c45e6700 1 --2- 172.21.15.149:0/4012112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd5bc0f5410 0x7fd5bc1067d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:38:56.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.696+0000 7fd5c45e6700 1 -- 172.21.15.149:0/4012112 >> 172.21.15.149:0/4012112 conn(0x7fd5bc0f0b70 msgr2=0x7fd5bc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:38:56.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.696+0000 7fd5c45e6700 1 -- 172.21.15.149:0/4012112 shutdown_connections 2024-02-12T16:38:56.699 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:38:56.696+0000 7fd5c45e6700 1 -- 172.21.15.149:0/4012112 wait complete. 2024-02-12T16:38:56.699 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:38:57.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:57 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:38:56.695349+0000 mon.smithi104 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.149:0/4012112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:38:58.417 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:38:58.418 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:38:58.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:38:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:56.363455+0000 mgr.smithi104.ohwbhf (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:00.269 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.266+0000 7f6b873bc700 1 -- 172.21.15.149:0/1837334354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b80060140 msgr2=0x7f6b800f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:00.270 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.266+0000 7f6b873bc700 1 --2- 172.21.15.149:0/1837334354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b80060140 0x7f6b800f8fc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f6b70009fb0 tx=0x7f6b70009b30).stop 2024-02-12T16:39:00.270 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.266+0000 7f6b873bc700 1 -- 172.21.15.149:0/1837334354 shutdown_connections 2024-02-12T16:39:00.270 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.266+0000 7f6b873bc700 1 --2- 172.21.15.149:0/1837334354 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b80060140 0x7f6b800f8fc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:00.270 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.266+0000 7f6b873bc700 1 -- 172.21.15.149:0/1837334354 >> 172.21.15.149:0/1837334354 conn(0x7f6b800f09a0 msgr2=0x7f6b800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:00.270 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.266+0000 7f6b873bc700 1 -- 172.21.15.149:0/1837334354 shutdown_connections 2024-02-12T16:39:00.270 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.266+0000 7f6b873bc700 1 -- 172.21.15.149:0/1837334354 wait complete. 2024-02-12T16:39:00.270 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.266+0000 7f6b873bc700 1 Processor -- start 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.266+0000 7f6b873bc700 1 -- start start 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b873bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b80060140 0x7f6b800f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b873bc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800f4ec0 con 0x7f6b80060140 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b863ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b80060140 0x7f6b800f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b863ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b80060140 0x7f6b800f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:59154/0 (socket says 172.21.15.149:59154) 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b863ba700 1 -- 172.21.15.149:0/2052347395 learned_addr learned my addr 172.21.15.149:0/2052347395 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b863ba700 1 -- 172.21.15.149:0/2052347395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b70005040 con 0x7f6b80060140 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b863ba700 1 --2- 172.21.15.149:0/2052347395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b80060140 0x7f6b800f6a40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f6b70005010 tx=0x7f6b7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b76ffd700 1 -- 172.21.15.149:0/2052347395 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b70009320 con 0x7f6b80060140 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b873bc700 1 -- 172.21.15.149:0/2052347395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b800f5060 con 0x7f6b80060140 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b76ffd700 1 -- 172.21.15.149:0/2052347395 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b70013380 con 0x7f6b80060140 2024-02-12T16:39:00.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b76ffd700 1 -- 172.21.15.149:0/2052347395 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b70014e50 con 0x7f6b80060140 2024-02-12T16:39:00.272 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.267+0000 7f6b873bc700 1 -- 172.21.15.149:0/2052347395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b800f54f0 con 0x7f6b80060140 2024-02-12T16:39:00.272 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.268+0000 7f6b76ffd700 1 -- 172.21.15.149:0/2052347395 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b700263a0 con 0x7f6b80060140 2024-02-12T16:39:00.272 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.268+0000 7f6b76ffd700 1 --2- 172.21.15.149:0/2052347395 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6b6c02ff10 0x7f6b6c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:00.273 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.268+0000 7f6b873bc700 1 -- 172.21.15.149:0/2052347395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b80059100 con 0x7f6b80060140 2024-02-12T16:39:00.273 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.268+0000 7f6b76ffd700 1 -- 172.21.15.149:0/2052347395 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b70041250 con 0x7f6b80060140 2024-02-12T16:39:00.273 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.269+0000 7f6b85bb9700 1 --2- 172.21.15.149:0/2052347395 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6b6c02ff10 0x7f6b6c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:00.273 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.269+0000 7f6b85bb9700 1 --2- 172.21.15.149:0/2052347395 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6b6c02ff10 0x7f6b6c0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f6b7c0096a0 tx=0x7f6b7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:00.275 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.273+0000 7f6b76ffd700 1 -- 172.21.15.149:0/2052347395 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b700126f0 con 0x7f6b80060140 2024-02-12T16:39:00.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.478+0000 7f6b873bc700 1 -- 172.21.15.149:0/2052347395 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b800fbe20 con 0x7f6b80060140 2024-02-12T16:39:00.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.478+0000 7f6b76ffd700 1 -- 172.21.15.149:0/2052347395 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6b70014420 con 0x7f6b80060140 2024-02-12T16:39:00.481 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:00.481 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:00.482 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.480+0000 7f6b873bc700 1 -- 172.21.15.149:0/2052347395 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6b6c02ff10 msgr2=0x7f6b6c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:00.482 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.480+0000 7f6b873bc700 1 --2- 172.21.15.149:0/2052347395 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6b6c02ff10 0x7f6b6c0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f6b7c0096a0 tx=0x7f6b7c006b40).stop 2024-02-12T16:39:00.482 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.480+0000 7f6b873bc700 1 -- 172.21.15.149:0/2052347395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b80060140 msgr2=0x7f6b800f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:00.483 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.480+0000 7f6b873bc700 1 --2- 172.21.15.149:0/2052347395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b80060140 0x7f6b800f6a40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f6b70005010 tx=0x7f6b7000b040).stop 2024-02-12T16:39:00.483 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.480+0000 7f6b873bc700 1 -- 172.21.15.149:0/2052347395 shutdown_connections 2024-02-12T16:39:00.483 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.480+0000 7f6b873bc700 1 --2- 172.21.15.149:0/2052347395 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6b6c02ff10 0x7f6b6c0323d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:00.483 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.480+0000 7f6b873bc700 1 --2- 172.21.15.149:0/2052347395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b80060140 0x7f6b800f6a40 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:00.483 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.480+0000 7f6b873bc700 1 -- 172.21.15.149:0/2052347395 >> 172.21.15.149:0/2052347395 conn(0x7f6b800f09a0 msgr2=0x7f6b800f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:00.483 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.480+0000 7f6b873bc700 1 -- 172.21.15.149:0/2052347395 shutdown_connections 2024-02-12T16:39:00.483 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:00.480+0000 7f6b873bc700 1 -- 172.21.15.149:0/2052347395 wait complete. 2024-02-12T16:39:00.483 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:00.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:38:58.363797+0000 mgr.smithi104.ohwbhf (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:01.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:00.479630+0000 mon.smithi104 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.149:0/2052347395' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:02.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:02.111 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:02.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:00.364176+0000 mgr.smithi104.ohwbhf (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:03.926 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.923+0000 7f41c2d3c700 1 -- 172.21.15.149:0/1483270195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41bc060220 msgr2=0x7f41bc0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:03.927 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.923+0000 7f41c2d3c700 1 --2- 172.21.15.149:0/1483270195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41bc060220 0x7f41bc0f9150 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f41b0009fb0 tx=0x7f41b0009b30).stop 2024-02-12T16:39:03.927 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.923+0000 7f41c2d3c700 1 -- 172.21.15.149:0/1483270195 shutdown_connections 2024-02-12T16:39:03.927 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.923+0000 7f41c2d3c700 1 --2- 172.21.15.149:0/1483270195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41bc060220 0x7f41bc0f9150 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:03.927 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.923+0000 7f41c2d3c700 1 -- 172.21.15.149:0/1483270195 >> 172.21.15.149:0/1483270195 conn(0x7f41bc0f0b70 msgr2=0x7f41bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:03.927 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.923+0000 7f41c2d3c700 1 -- 172.21.15.149:0/1483270195 shutdown_connections 2024-02-12T16:39:03.927 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.923+0000 7f41c2d3c700 1 -- 172.21.15.149:0/1483270195 wait complete. 2024-02-12T16:39:03.927 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.924+0000 7f41c2d3c700 1 Processor -- start 2024-02-12T16:39:03.927 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.924+0000 7f41c2d3c700 1 -- start start 2024-02-12T16:39:03.927 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.924+0000 7f41c2d3c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41bc060220 0x7f41bc0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:03.927 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.924+0000 7f41c2d3c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0f5050 con 0x7f41bc060220 2024-02-12T16:39:03.928 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.924+0000 7f41c0ad8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41bc060220 0x7f41bc0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:03.928 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.924+0000 7f41c0ad8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41bc060220 0x7f41bc0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:59174/0 (socket says 172.21.15.149:59174) 2024-02-12T16:39:03.928 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.924+0000 7f41c0ad8700 1 -- 172.21.15.149:0/3902489222 learned_addr learned my addr 172.21.15.149:0/3902489222 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:03.928 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.925+0000 7f41c0ad8700 1 -- 172.21.15.149:0/3902489222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b0005040 con 0x7f41bc060220 2024-02-12T16:39:03.928 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.925+0000 7f41c0ad8700 1 --2- 172.21.15.149:0/3902489222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41bc060220 0x7f41bc0f6bd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f41b00089f0 tx=0x7f41b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:03.929 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.925+0000 7f41b97fa700 1 -- 172.21.15.149:0/3902489222 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b0009320 con 0x7f41bc060220 2024-02-12T16:39:03.929 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.925+0000 7f41c2d3c700 1 -- 172.21.15.149:0/3902489222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41bc0f51f0 con 0x7f41bc060220 2024-02-12T16:39:03.929 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.925+0000 7f41b97fa700 1 -- 172.21.15.149:0/3902489222 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41b0013380 con 0x7f41bc060220 2024-02-12T16:39:03.929 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.925+0000 7f41b97fa700 1 -- 172.21.15.149:0/3902489222 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b0014e20 con 0x7f41bc060220 2024-02-12T16:39:03.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.925+0000 7f41c2d3c700 1 -- 172.21.15.149:0/3902489222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41bc0f5680 con 0x7f41bc060220 2024-02-12T16:39:03.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.926+0000 7f41b97fa700 1 -- 172.21.15.149:0/3902489222 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41b00263a0 con 0x7f41bc060220 2024-02-12T16:39:03.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.926+0000 7f41c2d3c700 1 -- 172.21.15.149:0/3902489222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41bc059100 con 0x7f41bc060220 2024-02-12T16:39:03.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.926+0000 7f41b97fa700 1 --2- 172.21.15.149:0/3902489222 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f41ac02ff10 0x7f41ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:03.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.926+0000 7f41b97fa700 1 -- 172.21.15.149:0/3902489222 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41b0040f00 con 0x7f41bc060220 2024-02-12T16:39:03.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.926+0000 7f41bbfff700 1 --2- 172.21.15.149:0/3902489222 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f41ac02ff10 0x7f41ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:03.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.927+0000 7f41bbfff700 1 --2- 172.21.15.149:0/3902489222 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f41ac02ff10 0x7f41ac0323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f41a80096a0 tx=0x7f41a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:03.933 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:03.931+0000 7f41b97fa700 1 -- 172.21.15.149:0/3902489222 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41b001db70 con 0x7f41bc060220 2024-02-12T16:39:04.138 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.136+0000 7f41c2d3c700 1 -- 172.21.15.149:0/3902489222 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41bc0fc090 con 0x7f41bc060220 2024-02-12T16:39:04.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.137+0000 7f41b97fa700 1 -- 172.21.15.149:0/3902489222 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f41b001db70 con 0x7f41bc060220 2024-02-12T16:39:04.139 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:04.139 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:04.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.139+0000 7f41c2d3c700 1 -- 172.21.15.149:0/3902489222 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f41ac02ff10 msgr2=0x7f41ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:04.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.139+0000 7f41c2d3c700 1 --2- 172.21.15.149:0/3902489222 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f41ac02ff10 0x7f41ac0323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f41a80096a0 tx=0x7f41a8006b40).stop 2024-02-12T16:39:04.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.139+0000 7f41c2d3c700 1 -- 172.21.15.149:0/3902489222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41bc060220 msgr2=0x7f41bc0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:04.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.139+0000 7f41c2d3c700 1 --2- 172.21.15.149:0/3902489222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41bc060220 0x7f41bc0f6bd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f41b00089f0 tx=0x7f41b000b040).stop 2024-02-12T16:39:04.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.139+0000 7f41c2d3c700 1 -- 172.21.15.149:0/3902489222 shutdown_connections 2024-02-12T16:39:04.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.139+0000 7f41c2d3c700 1 --2- 172.21.15.149:0/3902489222 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f41ac02ff10 0x7f41ac0323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:04.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.139+0000 7f41c2d3c700 1 --2- 172.21.15.149:0/3902489222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41bc060220 0x7f41bc0f6bd0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:04.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.139+0000 7f41c2d3c700 1 -- 172.21.15.149:0/3902489222 >> 172.21.15.149:0/3902489222 conn(0x7f41bc0f0b70 msgr2=0x7f41bc0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:04.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.139+0000 7f41c2d3c700 1 -- 172.21.15.149:0/3902489222 shutdown_connections 2024-02-12T16:39:04.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:04.139+0000 7f41c2d3c700 1 -- 172.21.15.149:0/3902489222 wait complete. 2024-02-12T16:39:04.142 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:04.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:02.364464+0000 mgr.smithi104.ohwbhf (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:04.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:04 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:04.138289+0000 mon.smithi104 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.149:0/3902489222' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:05.741 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:05.741 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:06.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:04.364790+0000 mgr.smithi104.ohwbhf (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:07.726 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.694+0000 7f36f5aa9700 1 -- 172.21.15.149:0/286263750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36f0060280 msgr2=0x7f36f00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:07.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.694+0000 7f36f5aa9700 1 --2- 172.21.15.149:0/286263750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36f0060280 0x7f36f00f90f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f36d8009fb0 tx=0x7f36d8009b30).stop 2024-02-12T16:39:07.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.694+0000 7f36f5aa9700 1 -- 172.21.15.149:0/286263750 shutdown_connections 2024-02-12T16:39:07.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.694+0000 7f36f5aa9700 1 --2- 172.21.15.149:0/286263750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36f0060280 0x7f36f00f90f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:07.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.694+0000 7f36f5aa9700 1 -- 172.21.15.149:0/286263750 >> 172.21.15.149:0/286263750 conn(0x7f36f00f0b70 msgr2=0x7f36f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:07.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.694+0000 7f36f5aa9700 1 -- 172.21.15.149:0/286263750 shutdown_connections 2024-02-12T16:39:07.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.694+0000 7f36f5aa9700 1 -- 172.21.15.149:0/286263750 wait complete. 2024-02-12T16:39:07.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.695+0000 7f36f5aa9700 1 Processor -- start 2024-02-12T16:39:07.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.695+0000 7f36f5aa9700 1 -- start start 2024-02-12T16:39:07.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.695+0000 7f36f5aa9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36f0060280 0x7f36f0106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:07.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.695+0000 7f36f5aa9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f0106c70 con 0x7f36f0060280 2024-02-12T16:39:07.727 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.695+0000 7f36ef7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36f0060280 0x7f36f0106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.695+0000 7f36ef7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36f0060280 0x7f36f0106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:39586/0 (socket says 172.21.15.149:39586) 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.695+0000 7f36ef7fe700 1 -- 172.21.15.149:0/1636964925 learned_addr learned my addr 172.21.15.149:0/1636964925 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.696+0000 7f36ef7fe700 1 -- 172.21.15.149:0/1636964925 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8005040 con 0x7f36f0060280 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.696+0000 7f36ef7fe700 1 --2- 172.21.15.149:0/1636964925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36f0060280 0x7f36f0106780 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f36d8009830 tx=0x7f36d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.696+0000 7f36e7fff700 1 -- 172.21.15.149:0/1636964925 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d8009320 con 0x7f36f0060280 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.696+0000 7f36f5aa9700 1 -- 172.21.15.149:0/1636964925 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f0106e70 con 0x7f36f0060280 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.696+0000 7f36e7fff700 1 -- 172.21.15.149:0/1636964925 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36d8013380 con 0x7f36f0060280 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.696+0000 7f36e7fff700 1 -- 172.21.15.149:0/1636964925 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d8014420 con 0x7f36f0060280 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.696+0000 7f36f5aa9700 1 -- 172.21.15.149:0/1636964925 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f0107300 con 0x7f36f0060280 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.697+0000 7f36e7fff700 1 -- 172.21.15.149:0/1636964925 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36d8014580 con 0x7f36f0060280 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.697+0000 7f36e7fff700 1 --2- 172.21.15.149:0/1636964925 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f36dc02ff60 0x7f36dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:07.728 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.697+0000 7f36e7fff700 1 -- 172.21.15.149:0/1636964925 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36d8040df0 con 0x7f36f0060280 2024-02-12T16:39:07.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.697+0000 7f36f5aa9700 1 -- 172.21.15.149:0/1636964925 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f0045ad0 con 0x7f36f0060280 2024-02-12T16:39:07.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.697+0000 7f36eeffd700 1 --2- 172.21.15.149:0/1636964925 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f36dc02ff60 0x7f36dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:07.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.698+0000 7f36eeffd700 1 --2- 172.21.15.149:0/1636964925 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f36dc02ff60 0x7f36dc032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f36e00096a0 tx=0x7f36e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:07.729 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.702+0000 7f36e7fff700 1 -- 172.21.15.149:0/1636964925 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36d80273f0 con 0x7f36f0060280 2024-02-12T16:39:07.910 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.908+0000 7f36f5aa9700 1 -- 172.21.15.149:0/1636964925 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36f0108040 con 0x7f36f0060280 2024-02-12T16:39:07.911 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.909+0000 7f36e7fff700 1 -- 172.21.15.149:0/1636964925 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f36d801c890 con 0x7f36f0060280 2024-02-12T16:39:07.911 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:07.911 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:07.913 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.910+0000 7f36f5aa9700 1 -- 172.21.15.149:0/1636964925 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f36dc02ff60 msgr2=0x7f36dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:07.913 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.910+0000 7f36f5aa9700 1 --2- 172.21.15.149:0/1636964925 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f36dc02ff60 0x7f36dc032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f36e00096a0 tx=0x7f36e0006b40).stop 2024-02-12T16:39:07.913 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.910+0000 7f36f5aa9700 1 -- 172.21.15.149:0/1636964925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36f0060280 msgr2=0x7f36f0106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:07.913 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.910+0000 7f36f5aa9700 1 --2- 172.21.15.149:0/1636964925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36f0060280 0x7f36f0106780 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f36d8009830 tx=0x7f36d800b040).stop 2024-02-12T16:39:07.913 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.910+0000 7f36f5aa9700 1 -- 172.21.15.149:0/1636964925 shutdown_connections 2024-02-12T16:39:07.913 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.910+0000 7f36f5aa9700 1 --2- 172.21.15.149:0/1636964925 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f36dc02ff60 0x7f36dc032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:07.913 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.910+0000 7f36f5aa9700 1 --2- 172.21.15.149:0/1636964925 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f36f0060280 0x7f36f0106780 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:07.913 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.910+0000 7f36f5aa9700 1 -- 172.21.15.149:0/1636964925 >> 172.21.15.149:0/1636964925 conn(0x7f36f00f0b70 msgr2=0x7f36f00f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:07.913 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.911+0000 7f36f5aa9700 1 -- 172.21.15.149:0/1636964925 shutdown_connections 2024-02-12T16:39:07.913 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:07.911+0000 7f36f5aa9700 1 -- 172.21.15.149:0/1636964925 wait complete. 2024-02-12T16:39:07.914 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:08.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:06.365173+0000 mgr.smithi104.ohwbhf (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:08.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:08 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:07.910167+0000 mon.smithi104 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.149:0/1636964925' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:09.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:09.662 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:10.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:10 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:08.365454+0000 mgr.smithi104.ohwbhf (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:11.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.560+0000 7f1e75268700 1 -- 172.21.15.149:0/2024981044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e700f6a10 msgr2=0x7f1e700f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:11.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.560+0000 7f1e75268700 1 --2- 172.21.15.149:0/2024981044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e700f6a10 0x7f1e700f6e30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f1e64009fb0 tx=0x7f1e64009b30).stop 2024-02-12T16:39:11.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.560+0000 7f1e75268700 1 -- 172.21.15.149:0/2024981044 shutdown_connections 2024-02-12T16:39:11.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.560+0000 7f1e75268700 1 --2- 172.21.15.149:0/2024981044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e700f6a10 0x7f1e700f6e30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:11.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.560+0000 7f1e75268700 1 -- 172.21.15.149:0/2024981044 >> 172.21.15.149:0/2024981044 conn(0x7f1e700f02e0 msgr2=0x7f1e700f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:11.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.560+0000 7f1e75268700 1 -- 172.21.15.149:0/2024981044 shutdown_connections 2024-02-12T16:39:11.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.560+0000 7f1e75268700 1 -- 172.21.15.149:0/2024981044 wait complete. 2024-02-12T16:39:11.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.561+0000 7f1e75268700 1 Processor -- start 2024-02-12T16:39:11.564 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.561+0000 7f1e75268700 1 -- start start 2024-02-12T16:39:11.564 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.561+0000 7f1e75268700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e700f6a10 0x7f1e70109490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:11.564 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.561+0000 7f1e75268700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e700f5350 con 0x7f1e700f6a10 2024-02-12T16:39:11.565 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.561+0000 7f1e6effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e700f6a10 0x7f1e70109490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.561+0000 7f1e6effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e700f6a10 0x7f1e70109490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:39602/0 (socket says 172.21.15.149:39602) 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.561+0000 7f1e6effd700 1 -- 172.21.15.149:0/1887200501 learned_addr learned my addr 172.21.15.149:0/1887200501 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.562+0000 7f1e6effd700 1 -- 172.21.15.149:0/1887200501 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e64005040 con 0x7f1e700f6a10 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.562+0000 7f1e6effd700 1 --2- 172.21.15.149:0/1887200501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e700f6a10 0x7f1e70109490 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f1e64005010 tx=0x7f1e6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.562+0000 7f1e577fe700 1 -- 172.21.15.149:0/1887200501 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e64009320 con 0x7f1e700f6a10 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.562+0000 7f1e75268700 1 -- 172.21.15.149:0/1887200501 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e700f5550 con 0x7f1e700f6a10 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.562+0000 7f1e577fe700 1 -- 172.21.15.149:0/1887200501 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e64013380 con 0x7f1e700f6a10 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.562+0000 7f1e577fe700 1 -- 172.21.15.149:0/1887200501 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e64014e50 con 0x7f1e700f6a10 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.562+0000 7f1e75268700 1 -- 172.21.15.149:0/1887200501 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e70109b40 con 0x7f1e700f6a10 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.563+0000 7f1e577fe700 1 -- 172.21.15.149:0/1887200501 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e640095f0 con 0x7f1e700f6a10 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.563+0000 7f1e75268700 1 -- 172.21.15.149:0/1887200501 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e70045ad0 con 0x7f1e700f6a10 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.563+0000 7f1e577fe700 1 --2- 172.21.15.149:0/1887200501 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1e5802ff10 0x7f1e580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:11.566 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.563+0000 7f1e577fe700 1 -- 172.21.15.149:0/1887200501 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e64040e70 con 0x7f1e700f6a10 2024-02-12T16:39:11.567 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.563+0000 7f1e6e7fc700 1 --2- 172.21.15.149:0/1887200501 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1e5802ff10 0x7f1e580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:11.567 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.564+0000 7f1e6e7fc700 1 --2- 172.21.15.149:0/1887200501 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1e5802ff10 0x7f1e580323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f1e600096a0 tx=0x7f1e60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:11.569 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.568+0000 7f1e577fe700 1 -- 172.21.15.149:0/1887200501 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e64014420 con 0x7f1e700f6a10 2024-02-12T16:39:11.773 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.771+0000 7f1e75268700 1 -- 172.21.15.149:0/1887200501 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e7010a360 con 0x7f1e700f6a10 2024-02-12T16:39:11.774 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.772+0000 7f1e577fe700 1 -- 172.21.15.149:0/1887200501 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1e64010230 con 0x7f1e700f6a10 2024-02-12T16:39:11.774 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:11.774 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:11.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.773+0000 7f1e75268700 1 -- 172.21.15.149:0/1887200501 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1e5802ff10 msgr2=0x7f1e580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:11.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.773+0000 7f1e75268700 1 --2- 172.21.15.149:0/1887200501 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1e5802ff10 0x7f1e580323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f1e600096a0 tx=0x7f1e60006b40).stop 2024-02-12T16:39:11.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.773+0000 7f1e75268700 1 -- 172.21.15.149:0/1887200501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e700f6a10 msgr2=0x7f1e70109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:11.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.773+0000 7f1e75268700 1 --2- 172.21.15.149:0/1887200501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e700f6a10 0x7f1e70109490 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f1e64005010 tx=0x7f1e6400b040).stop 2024-02-12T16:39:11.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.773+0000 7f1e75268700 1 -- 172.21.15.149:0/1887200501 shutdown_connections 2024-02-12T16:39:11.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.773+0000 7f1e75268700 1 --2- 172.21.15.149:0/1887200501 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f1e5802ff10 0x7f1e580323d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:11.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.773+0000 7f1e75268700 1 --2- 172.21.15.149:0/1887200501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e700f6a10 0x7f1e70109490 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:11.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.773+0000 7f1e75268700 1 -- 172.21.15.149:0/1887200501 >> 172.21.15.149:0/1887200501 conn(0x7f1e700f02e0 msgr2=0x7f1e700f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:11.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.773+0000 7f1e75268700 1 -- 172.21.15.149:0/1887200501 shutdown_connections 2024-02-12T16:39:11.776 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:11.773+0000 7f1e75268700 1 -- 172.21.15.149:0/1887200501 wait complete. 2024-02-12T16:39:11.777 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:12.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:10.365785+0000 mgr.smithi104.ohwbhf (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:12.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:12 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:11.773125+0000 mon.smithi104 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.149:0/1887200501' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:13.440 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:13.440 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:14.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:14 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:12.366166+0000 mgr.smithi104.ohwbhf (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:15.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.311+0000 7fb67ca0b700 1 -- 172.21.15.149:0/2108478894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6780f5410 msgr2=0x7fb6780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:15.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.311+0000 7fb67ca0b700 1 --2- 172.21.15.149:0/2108478894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6780f5410 0x7fb6780f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fb66c009fb0 tx=0x7fb66c009b30).stop 2024-02-12T16:39:15.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.311+0000 7fb67ca0b700 1 -- 172.21.15.149:0/2108478894 shutdown_connections 2024-02-12T16:39:15.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.311+0000 7fb67ca0b700 1 --2- 172.21.15.149:0/2108478894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6780f5410 0x7fb6780f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:15.328 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.311+0000 7fb67ca0b700 1 -- 172.21.15.149:0/2108478894 >> 172.21.15.149:0/2108478894 conn(0x7fb6780f0b70 msgr2=0x7fb6780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.311+0000 7fb67ca0b700 1 -- 172.21.15.149:0/2108478894 shutdown_connections 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.311+0000 7fb67ca0b700 1 -- 172.21.15.149:0/2108478894 wait complete. 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.312+0000 7fb67ca0b700 1 Processor -- start 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.312+0000 7fb67ca0b700 1 -- start start 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.312+0000 7fb67ca0b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6780f5410 0x7fb6781056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.312+0000 7fb67ca0b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb678105bd0 con 0x7fb6780f5410 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.312+0000 7fb67659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6780f5410 0x7fb6781056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.312+0000 7fb67659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6780f5410 0x7fb6781056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:59906/0 (socket says 172.21.15.149:59906) 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.312+0000 7fb67659c700 1 -- 172.21.15.149:0/1569034807 learned_addr learned my addr 172.21.15.149:0/1569034807 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.313+0000 7fb67659c700 1 -- 172.21.15.149:0/1569034807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c005040 con 0x7fb6780f5410 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.313+0000 7fb67659c700 1 --2- 172.21.15.149:0/1569034807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6780f5410 0x7fb6781056e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fb66c005010 tx=0x7fb66c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.313+0000 7fb666ffd700 1 -- 172.21.15.149:0/1569034807 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb66c009320 con 0x7fb6780f5410 2024-02-12T16:39:15.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.313+0000 7fb67ca0b700 1 -- 172.21.15.149:0/1569034807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb678105dd0 con 0x7fb6780f5410 2024-02-12T16:39:15.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.313+0000 7fb666ffd700 1 -- 172.21.15.149:0/1569034807 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb66c013380 con 0x7fb6780f5410 2024-02-12T16:39:15.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.313+0000 7fb666ffd700 1 -- 172.21.15.149:0/1569034807 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb66c014e50 con 0x7fb6780f5410 2024-02-12T16:39:15.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.313+0000 7fb67ca0b700 1 -- 172.21.15.149:0/1569034807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb678106260 con 0x7fb6780f5410 2024-02-12T16:39:15.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.314+0000 7fb666ffd700 1 -- 172.21.15.149:0/1569034807 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb66c0263a0 con 0x7fb6780f5410 2024-02-12T16:39:15.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.314+0000 7fb666ffd700 1 --2- 172.21.15.149:0/1569034807 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb66002ff10 0x7fb6600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:15.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.314+0000 7fb666ffd700 1 -- 172.21.15.149:0/1569034807 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb66c040e90 con 0x7fb6780f5410 2024-02-12T16:39:15.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.314+0000 7fb67ca0b700 1 -- 172.21.15.149:0/1569034807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb678045ad0 con 0x7fb6780f5410 2024-02-12T16:39:15.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.314+0000 7fb675d9b700 1 --2- 172.21.15.149:0/1569034807 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb66002ff10 0x7fb6600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:15.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.314+0000 7fb675d9b700 1 --2- 172.21.15.149:0/1569034807 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb66002ff10 0x7fb6600323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fb6680096a0 tx=0x7fb668006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:15.330 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.319+0000 7fb666ffd700 1 -- 172.21.15.149:0/1569034807 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb66c012920 con 0x7fb6780f5410 2024-02-12T16:39:15.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.525+0000 7fb67ca0b700 1 -- 172.21.15.149:0/1569034807 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6780fb090 con 0x7fb6780f5410 2024-02-12T16:39:15.528 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.526+0000 7fb666ffd700 1 -- 172.21.15.149:0/1569034807 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb66c00f340 con 0x7fb6780f5410 2024-02-12T16:39:15.529 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:15.529 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:15.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.528+0000 7fb67ca0b700 1 -- 172.21.15.149:0/1569034807 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb66002ff10 msgr2=0x7fb6600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:15.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.528+0000 7fb67ca0b700 1 --2- 172.21.15.149:0/1569034807 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb66002ff10 0x7fb6600323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fb6680096a0 tx=0x7fb668006b40).stop 2024-02-12T16:39:15.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.528+0000 7fb67ca0b700 1 -- 172.21.15.149:0/1569034807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6780f5410 msgr2=0x7fb6781056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:15.530 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.528+0000 7fb67ca0b700 1 --2- 172.21.15.149:0/1569034807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6780f5410 0x7fb6781056e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fb66c005010 tx=0x7fb66c00b040).stop 2024-02-12T16:39:15.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.528+0000 7fb67ca0b700 1 -- 172.21.15.149:0/1569034807 shutdown_connections 2024-02-12T16:39:15.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.528+0000 7fb67ca0b700 1 --2- 172.21.15.149:0/1569034807 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb66002ff10 0x7fb6600323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:15.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.528+0000 7fb67ca0b700 1 --2- 172.21.15.149:0/1569034807 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6780f5410 0x7fb6781056e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:15.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.528+0000 7fb67ca0b700 1 -- 172.21.15.149:0/1569034807 >> 172.21.15.149:0/1569034807 conn(0x7fb6780f0b70 msgr2=0x7fb6780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:15.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.528+0000 7fb67ca0b700 1 -- 172.21.15.149:0/1569034807 shutdown_connections 2024-02-12T16:39:15.531 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:15.528+0000 7fb67ca0b700 1 -- 172.21.15.149:0/1569034807 wait complete. 2024-02-12T16:39:15.531 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:16.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:14.366504+0000 mgr.smithi104.ohwbhf (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:16.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:16 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:15.527530+0000 mon.smithi104 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.149:0/1569034807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:17.152 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:17.152 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:18.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:18 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:16.366772+0000 mgr.smithi104.ohwbhf (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:18.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.959+0000 7f0d7a40b700 1 -- 172.21.15.149:0/299127346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d740f5410 msgr2=0x7f0d740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:18.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.959+0000 7f0d7a40b700 1 --2- 172.21.15.149:0/299127346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d740f5410 0x7f0d740f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f0d64009fb0 tx=0x7f0d64009b30).stop 2024-02-12T16:39:18.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.959+0000 7f0d7a40b700 1 -- 172.21.15.149:0/299127346 shutdown_connections 2024-02-12T16:39:18.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.959+0000 7f0d7a40b700 1 --2- 172.21.15.149:0/299127346 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d740f5410 0x7f0d740f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:18.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.959+0000 7f0d7a40b700 1 -- 172.21.15.149:0/299127346 >> 172.21.15.149:0/299127346 conn(0x7f0d740f0b70 msgr2=0x7f0d740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:18.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.959+0000 7f0d7a40b700 1 -- 172.21.15.149:0/299127346 shutdown_connections 2024-02-12T16:39:18.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.959+0000 7f0d7a40b700 1 -- 172.21.15.149:0/299127346 wait complete. 2024-02-12T16:39:18.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.960+0000 7f0d7a40b700 1 Processor -- start 2024-02-12T16:39:18.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.960+0000 7f0d7a40b700 1 -- start start 2024-02-12T16:39:18.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.960+0000 7f0d7a40b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d740f5410 0x7f0d741089b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:18.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.960+0000 7f0d7a40b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d74108ea0 con 0x7f0d740f5410 2024-02-12T16:39:18.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.960+0000 7f0d73fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d740f5410 0x7f0d741089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:18.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.961+0000 7f0d73fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d740f5410 0x7f0d741089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:59928/0 (socket says 172.21.15.149:59928) 2024-02-12T16:39:18.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.961+0000 7f0d73fff700 1 -- 172.21.15.149:0/2360122983 learned_addr learned my addr 172.21.15.149:0/2360122983 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:18.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.961+0000 7f0d73fff700 1 -- 172.21.15.149:0/2360122983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d64005040 con 0x7f0d740f5410 2024-02-12T16:39:18.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.961+0000 7f0d73fff700 1 --2- 172.21.15.149:0/2360122983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d740f5410 0x7f0d741089b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f0d64005010 tx=0x7f0d6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:18.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.961+0000 7f0d70ff9700 1 -- 172.21.15.149:0/2360122983 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d64009320 con 0x7f0d740f5410 2024-02-12T16:39:18.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.961+0000 7f0d70ff9700 1 -- 172.21.15.149:0/2360122983 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d64013380 con 0x7f0d740f5410 2024-02-12T16:39:18.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.961+0000 7f0d7a40b700 1 -- 172.21.15.149:0/2360122983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d741090a0 con 0x7f0d740f5410 2024-02-12T16:39:18.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.961+0000 7f0d70ff9700 1 -- 172.21.15.149:0/2360122983 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d64014e50 con 0x7f0d740f5410 2024-02-12T16:39:18.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.961+0000 7f0d7a40b700 1 -- 172.21.15.149:0/2360122983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d74109530 con 0x7f0d740f5410 2024-02-12T16:39:18.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.962+0000 7f0d70ff9700 1 -- 172.21.15.149:0/2360122983 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d640095f0 con 0x7f0d740f5410 2024-02-12T16:39:18.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.962+0000 7f0d7a40b700 1 -- 172.21.15.149:0/2360122983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d74045ad0 con 0x7f0d740f5410 2024-02-12T16:39:18.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.962+0000 7f0d70ff9700 1 --2- 172.21.15.149:0/2360122983 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0d5c02ff10 0x7f0d5c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:18.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.962+0000 7f0d70ff9700 1 -- 172.21.15.149:0/2360122983 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d64040fa0 con 0x7f0d740f5410 2024-02-12T16:39:18.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.962+0000 7f0d737fe700 1 --2- 172.21.15.149:0/2360122983 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0d5c02ff10 0x7f0d5c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:18.966 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.963+0000 7f0d737fe700 1 --2- 172.21.15.149:0/2360122983 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0d5c02ff10 0x7f0d5c0323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0d680096a0 tx=0x7f0d68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:18.969 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:18.967+0000 7f0d70ff9700 1 -- 172.21.15.149:0/2360122983 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d64014420 con 0x7f0d740f5410 2024-02-12T16:39:19.173 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.171+0000 7f0d7a40b700 1 -- 172.21.15.149:0/2360122983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d740fafa0 con 0x7f0d740f5410 2024-02-12T16:39:19.174 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.173+0000 7f0d70ff9700 1 -- 172.21.15.149:0/2360122983 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0d64010230 con 0x7f0d740f5410 2024-02-12T16:39:19.174 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:19.174 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:19.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.173+0000 7f0d7a40b700 1 -- 172.21.15.149:0/2360122983 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0d5c02ff10 msgr2=0x7f0d5c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:19.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.173+0000 7f0d7a40b700 1 --2- 172.21.15.149:0/2360122983 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0d5c02ff10 0x7f0d5c0323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0d680096a0 tx=0x7f0d68006b40).stop 2024-02-12T16:39:19.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.173+0000 7f0d7a40b700 1 -- 172.21.15.149:0/2360122983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d740f5410 msgr2=0x7f0d741089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:19.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.173+0000 7f0d7a40b700 1 --2- 172.21.15.149:0/2360122983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d740f5410 0x7f0d741089b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f0d64005010 tx=0x7f0d6400b040).stop 2024-02-12T16:39:19.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.173+0000 7f0d7a40b700 1 -- 172.21.15.149:0/2360122983 shutdown_connections 2024-02-12T16:39:19.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.173+0000 7f0d7a40b700 1 --2- 172.21.15.149:0/2360122983 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0d5c02ff10 0x7f0d5c0323d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:19.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.173+0000 7f0d7a40b700 1 --2- 172.21.15.149:0/2360122983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d740f5410 0x7f0d741089b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:19.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.173+0000 7f0d7a40b700 1 -- 172.21.15.149:0/2360122983 >> 172.21.15.149:0/2360122983 conn(0x7f0d740f0b70 msgr2=0x7f0d7405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:19.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.173+0000 7f0d7a40b700 1 -- 172.21.15.149:0/2360122983 shutdown_connections 2024-02-12T16:39:19.176 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:19.173+0000 7f0d7a40b700 1 -- 172.21.15.149:0/2360122983 wait complete. 2024-02-12T16:39:19.177 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:19.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:19 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:19.172939+0000 mon.smithi104 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.149:0/2360122983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:18.367018+0000 mgr.smithi104.ohwbhf (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:20.806 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:20.806 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:22.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:22 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:20.367337+0000 mgr.smithi104.ohwbhf (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:22.684 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.680+0000 7f153cf72700 1 -- 172.21.15.149:0/1428679707 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1538060280 msgr2=0x7f15380f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:22.684 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.680+0000 7f153cf72700 1 --2- 172.21.15.149:0/1428679707 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1538060280 0x7f15380f90f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f152c009fb0 tx=0x7f152c009b30).stop 2024-02-12T16:39:22.684 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.680+0000 7f153cf72700 1 -- 172.21.15.149:0/1428679707 shutdown_connections 2024-02-12T16:39:22.685 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.681+0000 7f153cf72700 1 --2- 172.21.15.149:0/1428679707 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1538060280 0x7f15380f90f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:22.685 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.681+0000 7f153cf72700 1 -- 172.21.15.149:0/1428679707 >> 172.21.15.149:0/1428679707 conn(0x7f15380f0b70 msgr2=0x7f15380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:22.685 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.681+0000 7f153cf72700 1 -- 172.21.15.149:0/1428679707 shutdown_connections 2024-02-12T16:39:22.685 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.681+0000 7f153cf72700 1 -- 172.21.15.149:0/1428679707 wait complete. 2024-02-12T16:39:22.685 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.681+0000 7f153cf72700 1 Processor -- start 2024-02-12T16:39:22.685 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.681+0000 7f153cf72700 1 -- start start 2024-02-12T16:39:22.685 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.681+0000 7f153cf72700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1538060280 0x7f1538106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:22.685 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.681+0000 7f153cf72700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1538106c70 con 0x7f1538060280 2024-02-12T16:39:22.685 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.681+0000 7f153659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1538060280 0x7f1538106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:22.685 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.681+0000 7f153659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1538060280 0x7f1538106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:59948/0 (socket says 172.21.15.149:59948) 2024-02-12T16:39:22.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.681+0000 7f153659c700 1 -- 172.21.15.149:0/3887009879 learned_addr learned my addr 172.21.15.149:0/3887009879 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:22.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.682+0000 7f153659c700 1 -- 172.21.15.149:0/3887009879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c005040 con 0x7f1538060280 2024-02-12T16:39:22.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.682+0000 7f153659c700 1 --2- 172.21.15.149:0/3887009879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1538060280 0x7f1538106780 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f152c0089f0 tx=0x7f152c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:22.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.682+0000 7f1526ffd700 1 -- 172.21.15.149:0/3887009879 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f152c009320 con 0x7f1538060280 2024-02-12T16:39:22.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.682+0000 7f153cf72700 1 -- 172.21.15.149:0/3887009879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1538106e70 con 0x7f1538060280 2024-02-12T16:39:22.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.682+0000 7f1526ffd700 1 -- 172.21.15.149:0/3887009879 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f152c013380 con 0x7f1538060280 2024-02-12T16:39:22.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.682+0000 7f1526ffd700 1 -- 172.21.15.149:0/3887009879 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f152c014e20 con 0x7f1538060280 2024-02-12T16:39:22.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.682+0000 7f153cf72700 1 -- 172.21.15.149:0/3887009879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1538107300 con 0x7f1538060280 2024-02-12T16:39:22.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.683+0000 7f1526ffd700 1 -- 172.21.15.149:0/3887009879 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f152c0263a0 con 0x7f1538060280 2024-02-12T16:39:22.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.683+0000 7f153cf72700 1 -- 172.21.15.149:0/3887009879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1538045ad0 con 0x7f1538060280 2024-02-12T16:39:22.686 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.683+0000 7f1526ffd700 1 --2- 172.21.15.149:0/3887009879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f152002ff10 0x7f15200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:22.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.683+0000 7f1526ffd700 1 -- 172.21.15.149:0/3887009879 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f152c040fe0 con 0x7f1538060280 2024-02-12T16:39:22.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.683+0000 7f1535d9b700 1 --2- 172.21.15.149:0/3887009879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f152002ff10 0x7f15200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:22.687 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.684+0000 7f1535d9b700 1 --2- 172.21.15.149:0/3887009879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f152002ff10 0x7f15200323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f15280096a0 tx=0x7f1528006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:22.688 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.688+0000 7f1526ffd700 1 -- 172.21.15.149:0/3887009879 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f152c012330 con 0x7f1538060280 2024-02-12T16:39:22.892 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.891+0000 7f153cf72700 1 -- 172.21.15.149:0/3887009879 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15381081b0 con 0x7f1538060280 2024-02-12T16:39:22.892 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.891+0000 7f1526ffd700 1 -- 172.21.15.149:0/3887009879 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f152c01db70 con 0x7f1538060280 2024-02-12T16:39:22.893 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:22.893 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:22.894 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.893+0000 7f153cf72700 1 -- 172.21.15.149:0/3887009879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f152002ff10 msgr2=0x7f15200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:22.894 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.893+0000 7f153cf72700 1 --2- 172.21.15.149:0/3887009879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f152002ff10 0x7f15200323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f15280096a0 tx=0x7f1528006b40).stop 2024-02-12T16:39:22.894 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.893+0000 7f153cf72700 1 -- 172.21.15.149:0/3887009879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1538060280 msgr2=0x7f1538106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:22.894 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.893+0000 7f153cf72700 1 --2- 172.21.15.149:0/3887009879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1538060280 0x7f1538106780 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f152c0089f0 tx=0x7f152c00b040).stop 2024-02-12T16:39:22.894 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.893+0000 7f153cf72700 1 -- 172.21.15.149:0/3887009879 shutdown_connections 2024-02-12T16:39:22.895 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.893+0000 7f153cf72700 1 --2- 172.21.15.149:0/3887009879 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f152002ff10 0x7f15200323d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:22.895 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.893+0000 7f153cf72700 1 --2- 172.21.15.149:0/3887009879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1538060280 0x7f1538106780 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:22.895 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.893+0000 7f153cf72700 1 -- 172.21.15.149:0/3887009879 >> 172.21.15.149:0/3887009879 conn(0x7f15380f0b70 msgr2=0x7f15380f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:22.895 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.893+0000 7f153cf72700 1 -- 172.21.15.149:0/3887009879 shutdown_connections 2024-02-12T16:39:22.895 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:22.893+0000 7f153cf72700 1 -- 172.21.15.149:0/3887009879 wait complete. 2024-02-12T16:39:22.895 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:23.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:23 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:22.891699+0000 mon.smithi104 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.149:0/3887009879' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:24.524 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:24.525 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:24.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:22.367601+0000 mgr.smithi104.ohwbhf (mgr.14162) 307 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:25.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:24.412821+0000 mon.smithi104 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:39:25.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:24.418078+0000 mon.smithi104 (mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:39:26.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.329+0000 7fb71f39d700 1 -- 172.21.15.149:0/2367629614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7180f51a0 msgr2=0x7fb7180f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:26.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.329+0000 7fb71f39d700 1 --2- 172.21.15.149:0/2367629614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7180f51a0 0x7fb7180f55c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fb708009fb0 tx=0x7fb708009b30).stop 2024-02-12T16:39:26.331 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.329+0000 7fb71f39d700 1 -- 172.21.15.149:0/2367629614 shutdown_connections 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.329+0000 7fb71f39d700 1 --2- 172.21.15.149:0/2367629614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7180f51a0 0x7fb7180f55c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.329+0000 7fb71f39d700 1 -- 172.21.15.149:0/2367629614 >> 172.21.15.149:0/2367629614 conn(0x7fb7180f09a0 msgr2=0x7fb7180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.329+0000 7fb71f39d700 1 -- 172.21.15.149:0/2367629614 shutdown_connections 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.329+0000 7fb71f39d700 1 -- 172.21.15.149:0/2367629614 wait complete. 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.329+0000 7fb71f39d700 1 Processor -- start 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.329+0000 7fb71f39d700 1 -- start start 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.329+0000 7fb71f39d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7180f51a0 0x7fb7180fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.329+0000 7fb71f39d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7180fb6f0 con 0x7fb7180f51a0 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.330+0000 7fb71e39b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7180f51a0 0x7fb7180fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.330+0000 7fb71e39b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7180f51a0 0x7fb7180fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50588/0 (socket says 172.21.15.149:50588) 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.330+0000 7fb71e39b700 1 -- 172.21.15.149:0/2127925058 learned_addr learned my addr 172.21.15.149:0/2127925058 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.330+0000 7fb71e39b700 1 -- 172.21.15.149:0/2127925058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb708005040 con 0x7fb7180f51a0 2024-02-12T16:39:26.332 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.330+0000 7fb71e39b700 1 --2- 172.21.15.149:0/2127925058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7180f51a0 0x7fb7180fd270 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fb7080089f0 tx=0x7fb70800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:26.333 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.330+0000 7fb70effd700 1 -- 172.21.15.149:0/2127925058 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb708009320 con 0x7fb7180f51a0 2024-02-12T16:39:26.333 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.330+0000 7fb71f39d700 1 -- 172.21.15.149:0/2127925058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7180fb890 con 0x7fb7180f51a0 2024-02-12T16:39:26.333 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.330+0000 7fb70effd700 1 -- 172.21.15.149:0/2127925058 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb708013380 con 0x7fb7180f51a0 2024-02-12T16:39:26.333 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.330+0000 7fb70effd700 1 -- 172.21.15.149:0/2127925058 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb708014e20 con 0x7fb7180f51a0 2024-02-12T16:39:26.333 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.330+0000 7fb71f39d700 1 -- 172.21.15.149:0/2127925058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7180fbd20 con 0x7fb7180f51a0 2024-02-12T16:39:26.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.331+0000 7fb70effd700 1 -- 172.21.15.149:0/2127925058 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7080095f0 con 0x7fb7180f51a0 2024-02-12T16:39:26.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.331+0000 7fb71f39d700 1 -- 172.21.15.149:0/2127925058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb718059100 con 0x7fb7180f51a0 2024-02-12T16:39:26.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.331+0000 7fb70effd700 1 --2- 172.21.15.149:0/2127925058 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb70402ff10 0x7fb7040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:26.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.331+0000 7fb70effd700 1 -- 172.21.15.149:0/2127925058 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb708040e90 con 0x7fb7180f51a0 2024-02-12T16:39:26.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.332+0000 7fb71db9a700 1 --2- 172.21.15.149:0/2127925058 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb70402ff10 0x7fb7040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:26.334 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.332+0000 7fb71db9a700 1 --2- 172.21.15.149:0/2127925058 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb70402ff10 0x7fb7040323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb7140096a0 tx=0x7fb714006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:26.337 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.336+0000 7fb70effd700 1 -- 172.21.15.149:0/2127925058 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb708014420 con 0x7fb7180f51a0 2024-02-12T16:39:26.538 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.537+0000 7fb71f39d700 1 -- 172.21.15.149:0/2127925058 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7180fad30 con 0x7fb7180f51a0 2024-02-12T16:39:26.539 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.538+0000 7fb70effd700 1 -- 172.21.15.149:0/2127925058 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb70801d980 con 0x7fb7180f51a0 2024-02-12T16:39:26.539 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:26.539 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:26.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.539+0000 7fb71f39d700 1 -- 172.21.15.149:0/2127925058 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb70402ff10 msgr2=0x7fb7040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:26.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.539+0000 7fb71f39d700 1 --2- 172.21.15.149:0/2127925058 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb70402ff10 0x7fb7040323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb7140096a0 tx=0x7fb714006b40).stop 2024-02-12T16:39:26.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.540+0000 7fb71f39d700 1 -- 172.21.15.149:0/2127925058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7180f51a0 msgr2=0x7fb7180fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:26.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.540+0000 7fb71f39d700 1 --2- 172.21.15.149:0/2127925058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7180f51a0 0x7fb7180fd270 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fb7080089f0 tx=0x7fb70800b040).stop 2024-02-12T16:39:26.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.540+0000 7fb71f39d700 1 -- 172.21.15.149:0/2127925058 shutdown_connections 2024-02-12T16:39:26.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.540+0000 7fb71f39d700 1 --2- 172.21.15.149:0/2127925058 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb70402ff10 0x7fb7040323d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:26.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.540+0000 7fb71f39d700 1 --2- 172.21.15.149:0/2127925058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7180f51a0 0x7fb7180fd270 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:26.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.540+0000 7fb71f39d700 1 -- 172.21.15.149:0/2127925058 >> 172.21.15.149:0/2127925058 conn(0x7fb7180f09a0 msgr2=0x7fb71805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:26.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.540+0000 7fb71f39d700 1 -- 172.21.15.149:0/2127925058 shutdown_connections 2024-02-12T16:39:26.541 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:26.540+0000 7fb71f39d700 1 -- 172.21.15.149:0/2127925058 wait complete. 2024-02-12T16:39:26.542 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:26.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:26 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:24.367815+0000 mgr.smithi104.ohwbhf (mgr.14162) 308 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:27.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:26.538239+0000 mon.smithi104 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.149:0/2127925058' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:28.160 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:28.160 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:28.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:26.368207+0000 mgr.smithi104.ohwbhf (mgr.14162) 309 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:30.020 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.015+0000 7fed1ea8b700 1 -- 172.21.15.149:0/2383221021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed180f5410 msgr2=0x7fed180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:30.020 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.015+0000 7fed1ea8b700 1 --2- 172.21.15.149:0/2383221021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed180f5410 0x7fed180f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fed0c009fb0 tx=0x7fed0c009b30).stop 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.015+0000 7fed1ea8b700 1 -- 172.21.15.149:0/2383221021 shutdown_connections 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.015+0000 7fed1ea8b700 1 --2- 172.21.15.149:0/2383221021 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed180f5410 0x7fed180f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.015+0000 7fed1ea8b700 1 -- 172.21.15.149:0/2383221021 >> 172.21.15.149:0/2383221021 conn(0x7fed180f0b70 msgr2=0x7fed180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.015+0000 7fed1ea8b700 1 -- 172.21.15.149:0/2383221021 shutdown_connections 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.015+0000 7fed1ea8b700 1 -- 172.21.15.149:0/2383221021 wait complete. 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.016+0000 7fed1ea8b700 1 Processor -- start 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.016+0000 7fed1ea8b700 1 -- start start 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.016+0000 7fed1ea8b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed180f5410 0x7fed181067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.016+0000 7fed1ea8b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed18106ca0 con 0x7fed180f5410 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.016+0000 7fed1c827700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed180f5410 0x7fed181067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.016+0000 7fed1c827700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed180f5410 0x7fed181067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50606/0 (socket says 172.21.15.149:50606) 2024-02-12T16:39:30.021 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.016+0000 7fed1c827700 1 -- 172.21.15.149:0/1838689553 learned_addr learned my addr 172.21.15.149:0/1838689553 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:30.022 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.016+0000 7fed1c827700 1 -- 172.21.15.149:0/1838689553 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c005040 con 0x7fed180f5410 2024-02-12T16:39:30.022 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.017+0000 7fed1c827700 1 --2- 172.21.15.149:0/1838689553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed180f5410 0x7fed181067b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fed0c0089f0 tx=0x7fed0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:30.022 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.017+0000 7fed157fa700 1 -- 172.21.15.149:0/1838689553 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed0c009320 con 0x7fed180f5410 2024-02-12T16:39:30.022 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.017+0000 7fed1ea8b700 1 -- 172.21.15.149:0/1838689553 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed18106ea0 con 0x7fed180f5410 2024-02-12T16:39:30.022 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.017+0000 7fed157fa700 1 -- 172.21.15.149:0/1838689553 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed0c013380 con 0x7fed180f5410 2024-02-12T16:39:30.022 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.017+0000 7fed157fa700 1 -- 172.21.15.149:0/1838689553 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed0c01ed80 con 0x7fed180f5410 2024-02-12T16:39:30.022 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.017+0000 7fed1ea8b700 1 -- 172.21.15.149:0/1838689553 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed18107330 con 0x7fed180f5410 2024-02-12T16:39:30.023 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.018+0000 7fed157fa700 1 -- 172.21.15.149:0/1838689553 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed0c014420 con 0x7fed180f5410 2024-02-12T16:39:30.023 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.018+0000 7fed157fa700 1 --2- 172.21.15.149:0/1838689553 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed0802ff60 0x7fed08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:30.023 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.018+0000 7fed157fa700 1 -- 172.21.15.149:0/1838689553 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed0c040e80 con 0x7fed180f5410 2024-02-12T16:39:30.023 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.018+0000 7fed1ea8b700 1 -- 172.21.15.149:0/1838689553 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed18045ad0 con 0x7fed180f5410 2024-02-12T16:39:30.023 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.018+0000 7fed17fff700 1 --2- 172.21.15.149:0/1838689553 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed0802ff60 0x7fed08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:30.023 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.018+0000 7fed17fff700 1 --2- 172.21.15.149:0/1838689553 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed0802ff60 0x7fed08032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fed040096a0 tx=0x7fed04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:30.025 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.023+0000 7fed157fa700 1 -- 172.21.15.149:0/1838689553 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed0c01c910 con 0x7fed180f5410 2024-02-12T16:39:30.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.227+0000 7fed1ea8b700 1 -- 172.21.15.149:0/1838689553 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed180fb090 con 0x7fed180f5410 2024-02-12T16:39:30.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.227+0000 7fed157fa700 1 -- 172.21.15.149:0/1838689553 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fed0c01c320 con 0x7fed180f5410 2024-02-12T16:39:30.230 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:30.230 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:30.231 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.229+0000 7fed1ea8b700 1 -- 172.21.15.149:0/1838689553 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed0802ff60 msgr2=0x7fed08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:30.231 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.229+0000 7fed1ea8b700 1 --2- 172.21.15.149:0/1838689553 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed0802ff60 0x7fed08032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fed040096a0 tx=0x7fed04006b40).stop 2024-02-12T16:39:30.232 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.229+0000 7fed1ea8b700 1 -- 172.21.15.149:0/1838689553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed180f5410 msgr2=0x7fed181067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:30.232 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.229+0000 7fed1ea8b700 1 --2- 172.21.15.149:0/1838689553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed180f5410 0x7fed181067b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fed0c0089f0 tx=0x7fed0c00b040).stop 2024-02-12T16:39:30.232 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.229+0000 7fed1ea8b700 1 -- 172.21.15.149:0/1838689553 shutdown_connections 2024-02-12T16:39:30.232 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.229+0000 7fed1ea8b700 1 --2- 172.21.15.149:0/1838689553 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed0802ff60 0x7fed08032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:30.232 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.229+0000 7fed1ea8b700 1 --2- 172.21.15.149:0/1838689553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed180f5410 0x7fed181067b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:30.232 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.229+0000 7fed1ea8b700 1 -- 172.21.15.149:0/1838689553 >> 172.21.15.149:0/1838689553 conn(0x7fed180f0b70 msgr2=0x7fed1805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:30.232 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.229+0000 7fed1ea8b700 1 -- 172.21.15.149:0/1838689553 shutdown_connections 2024-02-12T16:39:30.232 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:30.229+0000 7fed1ea8b700 1 -- 172.21.15.149:0/1838689553 wait complete. 2024-02-12T16:39:30.232 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:30.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:28.368535+0000 mgr.smithi104.ohwbhf (mgr.14162) 310 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:30.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:30 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:30.228767+0000 mon.smithi104 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.149:0/1838689553' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:31.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:31.883 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:32.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:30.368836+0000 mgr.smithi104.ohwbhf (mgr.14162) 311 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:33.707 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.703+0000 7ff157571700 1 -- 172.21.15.149:0/1522785312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1500f5410 msgr2=0x7ff1500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:33.707 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.703+0000 7ff157571700 1 --2- 172.21.15.149:0/1522785312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1500f5410 0x7ff1500f5830 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7ff140009fb0 tx=0x7ff140009b30).stop 2024-02-12T16:39:33.707 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.703+0000 7ff157571700 1 -- 172.21.15.149:0/1522785312 shutdown_connections 2024-02-12T16:39:33.707 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.703+0000 7ff157571700 1 --2- 172.21.15.149:0/1522785312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1500f5410 0x7ff1500f5830 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:33.707 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.703+0000 7ff157571700 1 -- 172.21.15.149:0/1522785312 >> 172.21.15.149:0/1522785312 conn(0x7ff1500f0b70 msgr2=0x7ff1500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:33.707 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.703+0000 7ff157571700 1 -- 172.21.15.149:0/1522785312 shutdown_connections 2024-02-12T16:39:33.707 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.703+0000 7ff157571700 1 -- 172.21.15.149:0/1522785312 wait complete. 2024-02-12T16:39:33.707 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.704+0000 7ff157571700 1 Processor -- start 2024-02-12T16:39:33.707 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.704+0000 7ff157571700 1 -- start start 2024-02-12T16:39:33.707 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.704+0000 7ff157571700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1500f5410 0x7ff1501067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:33.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.704+0000 7ff157571700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff150106cd0 con 0x7ff1500f5410 2024-02-12T16:39:33.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.704+0000 7ff15530d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1500f5410 0x7ff1501067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:33.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.704+0000 7ff15530d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1500f5410 0x7ff1501067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:50618/0 (socket says 172.21.15.149:50618) 2024-02-12T16:39:33.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.704+0000 7ff15530d700 1 -- 172.21.15.149:0/2138819964 learned_addr learned my addr 172.21.15.149:0/2138819964 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:33.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.704+0000 7ff15530d700 1 -- 172.21.15.149:0/2138819964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff140005040 con 0x7ff1500f5410 2024-02-12T16:39:33.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.705+0000 7ff15530d700 1 --2- 172.21.15.149:0/2138819964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1500f5410 0x7ff1501067e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7ff140005010 tx=0x7ff14000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:33.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.705+0000 7ff145ffb700 1 -- 172.21.15.149:0/2138819964 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff140009320 con 0x7ff1500f5410 2024-02-12T16:39:33.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.705+0000 7ff157571700 1 -- 172.21.15.149:0/2138819964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff150106ed0 con 0x7ff1500f5410 2024-02-12T16:39:33.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.705+0000 7ff145ffb700 1 -- 172.21.15.149:0/2138819964 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff140013380 con 0x7ff1500f5410 2024-02-12T16:39:33.708 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.705+0000 7ff145ffb700 1 -- 172.21.15.149:0/2138819964 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff140014e50 con 0x7ff1500f5410 2024-02-12T16:39:33.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.705+0000 7ff157571700 1 -- 172.21.15.149:0/2138819964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff150107360 con 0x7ff1500f5410 2024-02-12T16:39:33.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.706+0000 7ff145ffb700 1 -- 172.21.15.149:0/2138819964 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1400263a0 con 0x7ff1500f5410 2024-02-12T16:39:33.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.706+0000 7ff145ffb700 1 --2- 172.21.15.149:0/2138819964 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff13c02ff10 0x7ff13c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:33.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.706+0000 7ff157571700 1 -- 172.21.15.149:0/2138819964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff150045ad0 con 0x7ff1500f5410 2024-02-12T16:39:33.709 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.706+0000 7ff145ffb700 1 -- 172.21.15.149:0/2138819964 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff140040eb0 con 0x7ff1500f5410 2024-02-12T16:39:33.710 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.706+0000 7ff154b0c700 1 --2- 172.21.15.149:0/2138819964 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff13c02ff10 0x7ff13c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:33.710 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.707+0000 7ff154b0c700 1 --2- 172.21.15.149:0/2138819964 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff13c02ff10 0x7ff13c0323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff14c0096a0 tx=0x7ff14c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:33.712 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.711+0000 7ff145ffb700 1 -- 172.21.15.149:0/2138819964 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff140012920 con 0x7ff1500f5410 2024-02-12T16:39:33.916 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.913+0000 7ff157571700 1 -- 172.21.15.149:0/2138819964 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1500fb090 con 0x7ff1500f5410 2024-02-12T16:39:33.916 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.914+0000 7ff145ffb700 1 -- 172.21.15.149:0/2138819964 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff14000f340 con 0x7ff1500f5410 2024-02-12T16:39:33.916 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:33.917 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:33.918 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.916+0000 7ff157571700 1 -- 172.21.15.149:0/2138819964 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff13c02ff10 msgr2=0x7ff13c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:33.918 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.916+0000 7ff157571700 1 --2- 172.21.15.149:0/2138819964 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff13c02ff10 0x7ff13c0323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff14c0096a0 tx=0x7ff14c006b40).stop 2024-02-12T16:39:33.918 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.916+0000 7ff157571700 1 -- 172.21.15.149:0/2138819964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1500f5410 msgr2=0x7ff1501067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:33.918 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.916+0000 7ff157571700 1 --2- 172.21.15.149:0/2138819964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1500f5410 0x7ff1501067e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7ff140005010 tx=0x7ff14000b040).stop 2024-02-12T16:39:33.918 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.916+0000 7ff157571700 1 -- 172.21.15.149:0/2138819964 shutdown_connections 2024-02-12T16:39:33.918 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.916+0000 7ff157571700 1 --2- 172.21.15.149:0/2138819964 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff13c02ff10 0x7ff13c0323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:33.918 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.916+0000 7ff157571700 1 --2- 172.21.15.149:0/2138819964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1500f5410 0x7ff1501067e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:33.918 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.916+0000 7ff157571700 1 -- 172.21.15.149:0/2138819964 >> 172.21.15.149:0/2138819964 conn(0x7ff1500f0b70 msgr2=0x7ff15005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:33.918 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.916+0000 7ff157571700 1 -- 172.21.15.149:0/2138819964 shutdown_connections 2024-02-12T16:39:33.919 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:33.916+0000 7ff157571700 1 -- 172.21.15.149:0/2138819964 wait complete. 2024-02-12T16:39:33.919 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:34.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:32.369190+0000 mgr.smithi104.ohwbhf (mgr.14162) 312 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:34.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:34 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:33.915352+0000 mon.smithi104 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.149:0/2138819964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:35.607 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:35.607 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:36.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:34.369516+0000 mgr.smithi104.ohwbhf (mgr.14162) 313 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:37.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.484+0000 7f10d8bda700 1 -- 172.21.15.149:0/2175532844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10d40f51a0 msgr2=0x7f10d40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:37.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.484+0000 7f10d8bda700 1 --2- 172.21.15.149:0/2175532844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10d40f51a0 0x7f10d40f55c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f10c4009fb0 tx=0x7f10c4009b30).stop 2024-02-12T16:39:37.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.484+0000 7f10d8bda700 1 -- 172.21.15.149:0/2175532844 shutdown_connections 2024-02-12T16:39:37.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.484+0000 7f10d8bda700 1 --2- 172.21.15.149:0/2175532844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10d40f51a0 0x7f10d40f55c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:37.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.484+0000 7f10d8bda700 1 -- 172.21.15.149:0/2175532844 >> 172.21.15.149:0/2175532844 conn(0x7f10d40f09a0 msgr2=0x7f10d40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:37.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.484+0000 7f10d8bda700 1 -- 172.21.15.149:0/2175532844 shutdown_connections 2024-02-12T16:39:37.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.484+0000 7f10d8bda700 1 -- 172.21.15.149:0/2175532844 wait complete. 2024-02-12T16:39:37.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.484+0000 7f10d8bda700 1 Processor -- start 2024-02-12T16:39:37.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.485+0000 7f10d8bda700 1 -- start start 2024-02-12T16:39:37.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.485+0000 7f10d8bda700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10d40f51a0 0x7f10d41065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:37.488 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.485+0000 7f10d8bda700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d4106ac0 con 0x7f10d40f51a0 2024-02-12T16:39:37.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.485+0000 7f10d37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10d40f51a0 0x7f10d41065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:37.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.485+0000 7f10d37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10d40f51a0 0x7f10d41065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:37912/0 (socket says 172.21.15.149:37912) 2024-02-12T16:39:37.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.485+0000 7f10d37fe700 1 -- 172.21.15.149:0/3061714305 learned_addr learned my addr 172.21.15.149:0/3061714305 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:37.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.486+0000 7f10d37fe700 1 -- 172.21.15.149:0/3061714305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c4005040 con 0x7f10d40f51a0 2024-02-12T16:39:37.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.486+0000 7f10d37fe700 1 --2- 172.21.15.149:0/3061714305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10d40f51a0 0x7f10d41065d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f10c40089f0 tx=0x7f10c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:37.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.486+0000 7f10bbfff700 1 -- 172.21.15.149:0/3061714305 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10c4009320 con 0x7f10d40f51a0 2024-02-12T16:39:37.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.486+0000 7f10d8bda700 1 -- 172.21.15.149:0/3061714305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d4106cc0 con 0x7f10d40f51a0 2024-02-12T16:39:37.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.486+0000 7f10d8bda700 1 -- 172.21.15.149:0/3061714305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d4107150 con 0x7f10d40f51a0 2024-02-12T16:39:37.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.486+0000 7f10bbfff700 1 -- 172.21.15.149:0/3061714305 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10c4013380 con 0x7f10d40f51a0 2024-02-12T16:39:37.489 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.486+0000 7f10bbfff700 1 -- 172.21.15.149:0/3061714305 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10c4014e40 con 0x7f10d40f51a0 2024-02-12T16:39:37.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.487+0000 7f10bbfff700 1 -- 172.21.15.149:0/3061714305 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10c40263a0 con 0x7f10d40f51a0 2024-02-12T16:39:37.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.487+0000 7f10d8bda700 1 -- 172.21.15.149:0/3061714305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d4045ad0 con 0x7f10d40f51a0 2024-02-12T16:39:37.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.487+0000 7f10bbfff700 1 --2- 172.21.15.149:0/3061714305 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10bc02ff10 0x7f10bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:37.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.487+0000 7f10bbfff700 1 -- 172.21.15.149:0/3061714305 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10c4041170 con 0x7f10d40f51a0 2024-02-12T16:39:37.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.487+0000 7f10d2ffd700 1 --2- 172.21.15.149:0/3061714305 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10bc02ff10 0x7f10bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:37.490 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.488+0000 7f10d2ffd700 1 --2- 172.21.15.149:0/3061714305 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10bc02ff10 0x7f10bc0323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f10c80096a0 tx=0x7f10c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:37.494 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.492+0000 7f10bbfff700 1 -- 172.21.15.149:0/3061714305 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10c401c320 con 0x7f10d40f51a0 2024-02-12T16:39:37.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.695+0000 7f10d8bda700 1 -- 172.21.15.149:0/3061714305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10d40fad30 con 0x7f10d40f51a0 2024-02-12T16:39:37.698 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.696+0000 7f10bbfff700 1 -- 172.21.15.149:0/3061714305 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f10c4012b20 con 0x7f10d40f51a0 2024-02-12T16:39:37.699 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:37.699 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:37.700 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.698+0000 7f10d8bda700 1 -- 172.21.15.149:0/3061714305 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10bc02ff10 msgr2=0x7f10bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:37.700 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.698+0000 7f10d8bda700 1 --2- 172.21.15.149:0/3061714305 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10bc02ff10 0x7f10bc0323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f10c80096a0 tx=0x7f10c8006b40).stop 2024-02-12T16:39:37.700 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.698+0000 7f10d8bda700 1 -- 172.21.15.149:0/3061714305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10d40f51a0 msgr2=0x7f10d41065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:37.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.698+0000 7f10d8bda700 1 --2- 172.21.15.149:0/3061714305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10d40f51a0 0x7f10d41065d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f10c40089f0 tx=0x7f10c400b040).stop 2024-02-12T16:39:37.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.698+0000 7f10d8bda700 1 -- 172.21.15.149:0/3061714305 shutdown_connections 2024-02-12T16:39:37.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.698+0000 7f10d8bda700 1 --2- 172.21.15.149:0/3061714305 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f10bc02ff10 0x7f10bc0323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:37.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.698+0000 7f10d8bda700 1 --2- 172.21.15.149:0/3061714305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10d40f51a0 0x7f10d41065d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:37.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.698+0000 7f10d8bda700 1 -- 172.21.15.149:0/3061714305 >> 172.21.15.149:0/3061714305 conn(0x7f10d40f09a0 msgr2=0x7f10d405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:37.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.698+0000 7f10d8bda700 1 -- 172.21.15.149:0/3061714305 shutdown_connections 2024-02-12T16:39:37.701 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:37.698+0000 7f10d8bda700 1 -- 172.21.15.149:0/3061714305 wait complete. 2024-02-12T16:39:37.701 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:38.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:36.369856+0000 mgr.smithi104.ohwbhf (mgr.14162) 314 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:38.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:38 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:37.697515+0000 mon.smithi104 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.149:0/3061714305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:39.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:39.518 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:40.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:40 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:38.370169+0000 mgr.smithi104.ohwbhf (mgr.14162) 315 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:41.264 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.261+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/2558497655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3a80f51a0 msgr2=0x7fd3a80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:41.264 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.261+0000 7fd3ad6fc700 1 --2- 172.21.15.149:0/2558497655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3a80f51a0 0x7fd3a80f55c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fd398009fb0 tx=0x7fd398009b30).stop 2024-02-12T16:39:41.264 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.261+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/2558497655 shutdown_connections 2024-02-12T16:39:41.264 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.261+0000 7fd3ad6fc700 1 --2- 172.21.15.149:0/2558497655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3a80f51a0 0x7fd3a80f55c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:41.264 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.261+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/2558497655 >> 172.21.15.149:0/2558497655 conn(0x7fd3a80f09a0 msgr2=0x7fd3a80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:41.264 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.261+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/2558497655 shutdown_connections 2024-02-12T16:39:41.265 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.261+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/2558497655 wait complete. 2024-02-12T16:39:41.265 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.261+0000 7fd3ad6fc700 1 Processor -- start 2024-02-12T16:39:41.265 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.261+0000 7fd3ad6fc700 1 -- start start 2024-02-12T16:39:41.265 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.262+0000 7fd3ad6fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:41.265 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.262+0000 7fd3ad6fc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8106ac0 con 0x7fd3a80f51a0 2024-02-12T16:39:41.265 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.262+0000 7fd3a7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:41.265 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.262+0000 7fd3a7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:37926/0 (socket says 172.21.15.149:37926) 2024-02-12T16:39:41.265 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.262+0000 7fd3a7fff700 1 -- 172.21.15.149:0/924065062 learned_addr learned my addr 172.21.15.149:0/924065062 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:41.265 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.262+0000 7fd3a7fff700 1 -- 172.21.15.149:0/924065062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd398005040 con 0x7fd3a80f51a0 2024-02-12T16:39:41.265 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.262+0000 7fd3a7fff700 1 --2- 172.21.15.149:0/924065062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81065d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fd3980089f0 tx=0x7fd39800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:41.266 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.262+0000 7fd3a4ff9700 1 -- 172.21.15.149:0/924065062 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd398009320 con 0x7fd3a80f51a0 2024-02-12T16:39:41.266 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.262+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/924065062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a8106cc0 con 0x7fd3a80f51a0 2024-02-12T16:39:41.266 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.262+0000 7fd3a4ff9700 1 -- 172.21.15.149:0/924065062 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd398013380 con 0x7fd3a80f51a0 2024-02-12T16:39:41.266 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.262+0000 7fd3a4ff9700 1 -- 172.21.15.149:0/924065062 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd398014e20 con 0x7fd3a80f51a0 2024-02-12T16:39:41.266 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.263+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/924065062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a8107150 con 0x7fd3a80f51a0 2024-02-12T16:39:41.266 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.263+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/924065062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a8045ad0 con 0x7fd3a80f51a0 2024-02-12T16:39:41.267 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.265+0000 7fd3a4ff9700 1 -- 172.21.15.149:0/924065062 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3980095f0 con 0x7fd3a80f51a0 2024-02-12T16:39:41.267 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.265+0000 7fd3a4ff9700 1 --2- 172.21.15.149:0/924065062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd39402ff10 0x7fd3940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:41.267 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.265+0000 7fd3a4ff9700 1 -- 172.21.15.149:0/924065062 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd398041020 con 0x7fd3a80f51a0 2024-02-12T16:39:41.267 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.265+0000 7fd3a77fe700 1 --2- 172.21.15.149:0/924065062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd39402ff10 0x7fd3940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:41.268 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.266+0000 7fd3a77fe700 1 --2- 172.21.15.149:0/924065062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd39402ff10 0x7fd3940323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd3900096a0 tx=0x7fd390006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:41.271 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.269+0000 7fd3a4ff9700 1 -- 172.21.15.149:0/924065062 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd39800f990 con 0x7fd3a80f51a0 2024-02-12T16:39:41.477 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.475+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/924065062 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3a80fad30 con 0x7fd3a80f51a0 2024-02-12T16:39:41.478 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.476+0000 7fd3a4ff9700 1 -- 172.21.15.149:0/924065062 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd39800f990 con 0x7fd3a80f51a0 2024-02-12T16:39:41.478 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:41.478 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:41.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.478+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/924065062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd39402ff10 msgr2=0x7fd3940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:41.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.478+0000 7fd3ad6fc700 1 --2- 172.21.15.149:0/924065062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd39402ff10 0x7fd3940323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd3900096a0 tx=0x7fd390006b40).stop 2024-02-12T16:39:41.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.478+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/924065062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3a80f51a0 msgr2=0x7fd3a81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:41.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.478+0000 7fd3ad6fc700 1 --2- 172.21.15.149:0/924065062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81065d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fd3980089f0 tx=0x7fd39800b040).stop 2024-02-12T16:39:41.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.478+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/924065062 shutdown_connections 2024-02-12T16:39:41.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.478+0000 7fd3ad6fc700 1 --2- 172.21.15.149:0/924065062 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fd39402ff10 0x7fd3940323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:41.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.478+0000 7fd3ad6fc700 1 --2- 172.21.15.149:0/924065062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81065d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:41.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.478+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/924065062 >> 172.21.15.149:0/924065062 conn(0x7fd3a80f09a0 msgr2=0x7fd3a805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:41.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.478+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/924065062 shutdown_connections 2024-02-12T16:39:41.480 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:41.478+0000 7fd3ad6fc700 1 -- 172.21.15.149:0/924065062 wait complete. 2024-02-12T16:39:41.481 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:42.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:40.370471+0000 mgr.smithi104.ohwbhf (mgr.14162) 316 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:42.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:42 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:41.477339+0000 mon.smithi104 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.149:0/924065062' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:43.227 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:43.227 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:44.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:44 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:42.370755+0000 mgr.smithi104.ohwbhf (mgr.14162) 317 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:45.116 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.113+0000 7f68f5b5c700 1 -- 172.21.15.149:0/1246543620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68f0060280 msgr2=0x7f68f00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:45.116 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.113+0000 7f68f5b5c700 1 --2- 172.21.15.149:0/1246543620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68f0060280 0x7f68f00f90f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f68e4009fb0 tx=0x7f68e4009b30).stop 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.113+0000 7f68f5b5c700 1 -- 172.21.15.149:0/1246543620 shutdown_connections 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.113+0000 7f68f5b5c700 1 --2- 172.21.15.149:0/1246543620 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68f0060280 0x7f68f00f90f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.113+0000 7f68f5b5c700 1 -- 172.21.15.149:0/1246543620 >> 172.21.15.149:0/1246543620 conn(0x7f68f00f0b70 msgr2=0x7f68f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.113+0000 7f68f5b5c700 1 -- 172.21.15.149:0/1246543620 shutdown_connections 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.113+0000 7f68f5b5c700 1 -- 172.21.15.149:0/1246543620 wait complete. 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.114+0000 7f68f5b5c700 1 Processor -- start 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.114+0000 7f68f5b5c700 1 -- start start 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.114+0000 7f68f5b5c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68f0060280 0x7f68f00f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.114+0000 7f68f5b5c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f00f4ff0 con 0x7f68f0060280 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.114+0000 7f68ef7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68f0060280 0x7f68f00f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.114+0000 7f68ef7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68f0060280 0x7f68f00f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:57778/0 (socket says 172.21.15.149:57778) 2024-02-12T16:39:45.117 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.114+0000 7f68ef7fe700 1 -- 172.21.15.149:0/2858356171 learned_addr learned my addr 172.21.15.149:0/2858356171 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:45.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.115+0000 7f68ef7fe700 1 -- 172.21.15.149:0/2858356171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e4005040 con 0x7f68f0060280 2024-02-12T16:39:45.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.115+0000 7f68ef7fe700 1 --2- 172.21.15.149:0/2858356171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68f0060280 0x7f68f00f6b70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f68e4009830 tx=0x7f68e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:45.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.115+0000 7f68d7fff700 1 -- 172.21.15.149:0/2858356171 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e4009320 con 0x7f68f0060280 2024-02-12T16:39:45.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.115+0000 7f68f5b5c700 1 -- 172.21.15.149:0/2858356171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f00f5190 con 0x7f68f0060280 2024-02-12T16:39:45.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.115+0000 7f68d7fff700 1 -- 172.21.15.149:0/2858356171 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68e4013380 con 0x7f68f0060280 2024-02-12T16:39:45.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.115+0000 7f68d7fff700 1 -- 172.21.15.149:0/2858356171 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68e4014420 con 0x7f68f0060280 2024-02-12T16:39:45.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.115+0000 7f68f5b5c700 1 -- 172.21.15.149:0/2858356171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f00f5620 con 0x7f68f0060280 2024-02-12T16:39:45.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.116+0000 7f68d7fff700 1 -- 172.21.15.149:0/2858356171 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68e401e420 con 0x7f68f0060280 2024-02-12T16:39:45.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.116+0000 7f68f5b5c700 1 -- 172.21.15.149:0/2858356171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f0059100 con 0x7f68f0060280 2024-02-12T16:39:45.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.116+0000 7f68d7fff700 1 --2- 172.21.15.149:0/2858356171 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f68d802ff60 0x7f68d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:45.119 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.116+0000 7f68d7fff700 1 -- 172.21.15.149:0/2858356171 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68e4040ef0 con 0x7f68f0060280 2024-02-12T16:39:45.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.116+0000 7f68eeffd700 1 --2- 172.21.15.149:0/2858356171 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f68d802ff60 0x7f68d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:45.120 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.117+0000 7f68eeffd700 1 --2- 172.21.15.149:0/2858356171 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f68d802ff60 0x7f68d8032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f68e00096a0 tx=0x7f68e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:45.123 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.121+0000 7f68d7fff700 1 -- 172.21.15.149:0/2858356171 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68e401cb10 con 0x7f68f0060280 2024-02-12T16:39:45.326 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.324+0000 7f68f5b5c700 1 -- 172.21.15.149:0/2858356171 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68f0045ad0 con 0x7f68f0060280 2024-02-12T16:39:45.327 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.325+0000 7f68d7fff700 1 -- 172.21.15.149:0/2858356171 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f68e4004490 con 0x7f68f0060280 2024-02-12T16:39:45.327 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:45.328 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:45.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.327+0000 7f68f5b5c700 1 -- 172.21.15.149:0/2858356171 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f68d802ff60 msgr2=0x7f68d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:45.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.327+0000 7f68f5b5c700 1 --2- 172.21.15.149:0/2858356171 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f68d802ff60 0x7f68d8032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f68e00096a0 tx=0x7f68e0006b40).stop 2024-02-12T16:39:45.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.327+0000 7f68f5b5c700 1 -- 172.21.15.149:0/2858356171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68f0060280 msgr2=0x7f68f00f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:45.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.327+0000 7f68f5b5c700 1 --2- 172.21.15.149:0/2858356171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68f0060280 0x7f68f00f6b70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f68e4009830 tx=0x7f68e400b040).stop 2024-02-12T16:39:45.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.327+0000 7f68f5b5c700 1 -- 172.21.15.149:0/2858356171 shutdown_connections 2024-02-12T16:39:45.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.327+0000 7f68f5b5c700 1 --2- 172.21.15.149:0/2858356171 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f68d802ff60 0x7f68d8032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:45.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.327+0000 7f68f5b5c700 1 --2- 172.21.15.149:0/2858356171 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68f0060280 0x7f68f00f6b70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:45.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.327+0000 7f68f5b5c700 1 -- 172.21.15.149:0/2858356171 >> 172.21.15.149:0/2858356171 conn(0x7f68f00f0b70 msgr2=0x7f68f00f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:45.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.327+0000 7f68f5b5c700 1 -- 172.21.15.149:0/2858356171 shutdown_connections 2024-02-12T16:39:45.329 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:45.327+0000 7f68f5b5c700 1 -- 172.21.15.149:0/2858356171 wait complete. 2024-02-12T16:39:45.330 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:46.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:44.371067+0000 mgr.smithi104.ohwbhf (mgr.14162) 318 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:46.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:46 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:45.326446+0000 mon.smithi104 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.149:0/2858356171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:47.058 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:47.058 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:48.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:48 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:46.371478+0000 mgr.smithi104.ohwbhf (mgr.14162) 319 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:49.110 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.106+0000 7f972a315700 1 -- 172.21.15.149:0/50703361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97240f5410 msgr2=0x7f97240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:49.110 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.106+0000 7f972a315700 1 --2- 172.21.15.149:0/50703361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97240f5410 0x7f97240f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f9714009fb0 tx=0x7f9714009b30).stop 2024-02-12T16:39:49.110 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.106+0000 7f972a315700 1 -- 172.21.15.149:0/50703361 shutdown_connections 2024-02-12T16:39:49.110 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.106+0000 7f972a315700 1 --2- 172.21.15.149:0/50703361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97240f5410 0x7f97240f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:49.110 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.106+0000 7f972a315700 1 -- 172.21.15.149:0/50703361 >> 172.21.15.149:0/50703361 conn(0x7f97240f0b70 msgr2=0x7f97240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.106+0000 7f972a315700 1 -- 172.21.15.149:0/50703361 shutdown_connections 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.106+0000 7f972a315700 1 -- 172.21.15.149:0/50703361 wait complete. 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.107+0000 7f972a315700 1 Processor -- start 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.107+0000 7f972a315700 1 -- start start 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.107+0000 7f972a315700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97240f5410 0x7f97240fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.107+0000 7f972a315700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97240fb960 con 0x7f97240f5410 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.107+0000 7f9723fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97240f5410 0x7f97240fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.107+0000 7f9723fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97240f5410 0x7f97240fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:57786/0 (socket says 172.21.15.149:57786) 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.107+0000 7f9723fff700 1 -- 172.21.15.149:0/1388883920 learned_addr learned my addr 172.21.15.149:0/1388883920 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.108+0000 7f9723fff700 1 -- 172.21.15.149:0/1388883920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714005040 con 0x7f97240f5410 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.108+0000 7f9723fff700 1 --2- 172.21.15.149:0/1388883920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97240f5410 0x7f97240fd4e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f9714005010 tx=0x7f971400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.108+0000 7f9720ff9700 1 -- 172.21.15.149:0/1388883920 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9714009320 con 0x7f97240f5410 2024-02-12T16:39:49.111 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.108+0000 7f9720ff9700 1 -- 172.21.15.149:0/1388883920 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9714013380 con 0x7f97240f5410 2024-02-12T16:39:49.112 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.108+0000 7f972a315700 1 -- 172.21.15.149:0/1388883920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97240fbb00 con 0x7f97240f5410 2024-02-12T16:39:49.112 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.108+0000 7f9720ff9700 1 -- 172.21.15.149:0/1388883920 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9714014e50 con 0x7f97240f5410 2024-02-12T16:39:49.112 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.108+0000 7f972a315700 1 -- 172.21.15.149:0/1388883920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97240fbf90 con 0x7f97240f5410 2024-02-12T16:39:49.113 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.109+0000 7f9720ff9700 1 -- 172.21.15.149:0/1388883920 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97140095f0 con 0x7f97240f5410 2024-02-12T16:39:49.113 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.109+0000 7f972a315700 1 -- 172.21.15.149:0/1388883920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9724059100 con 0x7f97240f5410 2024-02-12T16:39:49.113 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.109+0000 7f9720ff9700 1 --2- 172.21.15.149:0/1388883920 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f970c02ff10 0x7f970c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:49.113 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.109+0000 7f9720ff9700 1 -- 172.21.15.149:0/1388883920 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9714040f10 con 0x7f97240f5410 2024-02-12T16:39:49.113 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.109+0000 7f97237fe700 1 --2- 172.21.15.149:0/1388883920 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f970c02ff10 0x7f970c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:49.113 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.109+0000 7f97237fe700 1 --2- 172.21.15.149:0/1388883920 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f970c02ff10 0x7f970c0323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f97180096a0 tx=0x7f9718006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:49.116 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.114+0000 7f9720ff9700 1 -- 172.21.15.149:0/1388883920 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9714014420 con 0x7f97240f5410 2024-02-12T16:39:49.319 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.317+0000 7f972a315700 1 -- 172.21.15.149:0/1388883920 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97240fafa0 con 0x7f97240f5410 2024-02-12T16:39:49.320 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.318+0000 7f9720ff9700 1 -- 172.21.15.149:0/1388883920 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9714010230 con 0x7f97240f5410 2024-02-12T16:39:49.320 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:49.321 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:49.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.320+0000 7f972a315700 1 -- 172.21.15.149:0/1388883920 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f970c02ff10 msgr2=0x7f970c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:49.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.320+0000 7f972a315700 1 --2- 172.21.15.149:0/1388883920 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f970c02ff10 0x7f970c0323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f97180096a0 tx=0x7f9718006b40).stop 2024-02-12T16:39:49.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.320+0000 7f972a315700 1 -- 172.21.15.149:0/1388883920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97240f5410 msgr2=0x7f97240fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:49.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.320+0000 7f972a315700 1 --2- 172.21.15.149:0/1388883920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97240f5410 0x7f97240fd4e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f9714005010 tx=0x7f971400b040).stop 2024-02-12T16:39:49.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.320+0000 7f972a315700 1 -- 172.21.15.149:0/1388883920 shutdown_connections 2024-02-12T16:39:49.322 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.320+0000 7f972a315700 1 --2- 172.21.15.149:0/1388883920 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f970c02ff10 0x7f970c0323d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:49.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.320+0000 7f972a315700 1 --2- 172.21.15.149:0/1388883920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97240f5410 0x7f97240fd4e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:49.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.320+0000 7f972a315700 1 -- 172.21.15.149:0/1388883920 >> 172.21.15.149:0/1388883920 conn(0x7f97240f0b70 msgr2=0x7f972405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:49.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.320+0000 7f972a315700 1 -- 172.21.15.149:0/1388883920 shutdown_connections 2024-02-12T16:39:49.323 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:49.320+0000 7f972a315700 1 -- 172.21.15.149:0/1388883920 wait complete. 2024-02-12T16:39:49.323 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:50.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:48.371821+0000 mgr.smithi104.ohwbhf (mgr.14162) 320 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:50.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:49.319437+0000 mon.smithi104 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.149:0/1388883920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:50.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:50.836 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:52.587 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.584+0000 7f293861f700 1 -- 172.21.15.149:0/765537334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29300f6510 msgr2=0x7f29300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:52.587 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.584+0000 7f293861f700 1 --2- 172.21.15.149:0/765537334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29300f6510 0x7f29300f6930 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f2920009fb0 tx=0x7f2920009b30).stop 2024-02-12T16:39:52.587 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.584+0000 7f293861f700 1 -- 172.21.15.149:0/765537334 shutdown_connections 2024-02-12T16:39:52.587 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.584+0000 7f293861f700 1 --2- 172.21.15.149:0/765537334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29300f6510 0x7f29300f6930 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:52.587 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.584+0000 7f293861f700 1 -- 172.21.15.149:0/765537334 >> 172.21.15.149:0/765537334 conn(0x7f29300f1c70 msgr2=0x7f29300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.584+0000 7f293861f700 1 -- 172.21.15.149:0/765537334 shutdown_connections 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.584+0000 7f293861f700 1 -- 172.21.15.149:0/765537334 wait complete. 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.584+0000 7f293861f700 1 Processor -- start 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.584+0000 7f293861f700 1 -- start start 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.584+0000 7f293861f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29300f6510 0x7f29301067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.584+0000 7f293861f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2930106cd0 con 0x7f29300f6510 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.585+0000 7f29363bb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29300f6510 0x7f29301067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.585+0000 7f29363bb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29300f6510 0x7f29301067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:57808/0 (socket says 172.21.15.149:57808) 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.585+0000 7f29363bb700 1 -- 172.21.15.149:0/3673523060 learned_addr learned my addr 172.21.15.149:0/3673523060 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.585+0000 7f29363bb700 1 -- 172.21.15.149:0/3673523060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2920005040 con 0x7f29300f6510 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.585+0000 7f29363bb700 1 --2- 172.21.15.149:0/3673523060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29300f6510 0x7f29301067e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f2920005010 tx=0x7f292000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.585+0000 7f2926ffd700 1 -- 172.21.15.149:0/3673523060 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2920009320 con 0x7f29300f6510 2024-02-12T16:39:52.588 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.585+0000 7f2926ffd700 1 -- 172.21.15.149:0/3673523060 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2920013380 con 0x7f29300f6510 2024-02-12T16:39:52.589 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.585+0000 7f293861f700 1 -- 172.21.15.149:0/3673523060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2930106ed0 con 0x7f29300f6510 2024-02-12T16:39:52.589 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.585+0000 7f2926ffd700 1 -- 172.21.15.149:0/3673523060 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2920014e50 con 0x7f29300f6510 2024-02-12T16:39:52.589 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.585+0000 7f293861f700 1 -- 172.21.15.149:0/3673523060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2930107360 con 0x7f29300f6510 2024-02-12T16:39:52.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.586+0000 7f2926ffd700 1 -- 172.21.15.149:0/3673523060 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29200263a0 con 0x7f29300f6510 2024-02-12T16:39:52.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.586+0000 7f293861f700 1 -- 172.21.15.149:0/3673523060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2930045ad0 con 0x7f29300f6510 2024-02-12T16:39:52.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.586+0000 7f2926ffd700 1 --2- 172.21.15.149:0/3673523060 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f291c02ff10 0x7f291c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:52.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.586+0000 7f2926ffd700 1 -- 172.21.15.149:0/3673523060 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2920041250 con 0x7f29300f6510 2024-02-12T16:39:52.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.586+0000 7f2935bba700 1 --2- 172.21.15.149:0/3673523060 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f291c02ff10 0x7f291c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:52.590 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.587+0000 7f2935bba700 1 --2- 172.21.15.149:0/3673523060 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f291c02ff10 0x7f291c0323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f292c0096a0 tx=0x7f292c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:52.592 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.591+0000 7f2926ffd700 1 -- 172.21.15.149:0/3673523060 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f292001cb10 con 0x7f29300f6510 2024-02-12T16:39:52.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:52 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:50.372084+0000 mgr.smithi104.ohwbhf (mgr.14162) 321 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:52.795 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.793+0000 7f293861f700 1 -- 172.21.15.149:0/3673523060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2930108210 con 0x7f29300f6510 2024-02-12T16:39:52.795 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.793+0000 7f2926ffd700 1 -- 172.21.15.149:0/3673523060 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f292001cb10 con 0x7f29300f6510 2024-02-12T16:39:52.795 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:52.796 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:52.797 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.795+0000 7f293861f700 1 -- 172.21.15.149:0/3673523060 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f291c02ff10 msgr2=0x7f291c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:52.797 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.795+0000 7f293861f700 1 --2- 172.21.15.149:0/3673523060 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f291c02ff10 0x7f291c0323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f292c0096a0 tx=0x7f292c006b40).stop 2024-02-12T16:39:52.797 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.795+0000 7f293861f700 1 -- 172.21.15.149:0/3673523060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29300f6510 msgr2=0x7f29301067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:52.797 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.795+0000 7f293861f700 1 --2- 172.21.15.149:0/3673523060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29300f6510 0x7f29301067e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f2920005010 tx=0x7f292000b040).stop 2024-02-12T16:39:52.797 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.795+0000 7f293861f700 1 -- 172.21.15.149:0/3673523060 shutdown_connections 2024-02-12T16:39:52.798 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.795+0000 7f293861f700 1 --2- 172.21.15.149:0/3673523060 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f291c02ff10 0x7f291c0323d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:52.798 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.795+0000 7f293861f700 1 --2- 172.21.15.149:0/3673523060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29300f6510 0x7f29301067e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:52.798 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.795+0000 7f293861f700 1 -- 172.21.15.149:0/3673523060 >> 172.21.15.149:0/3673523060 conn(0x7f29300f1c70 msgr2=0x7f29300f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:52.798 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.795+0000 7f293861f700 1 -- 172.21.15.149:0/3673523060 shutdown_connections 2024-02-12T16:39:52.798 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:52.795+0000 7f293861f700 1 -- 172.21.15.149:0/3673523060 wait complete. 2024-02-12T16:39:52.798 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:53.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:53 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:52.794524+0000 mon.smithi104 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.149:0/3673523060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:54.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:54.530 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:54.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:52.372408+0000 mgr.smithi104.ohwbhf (mgr.14162) 322 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:56.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.299+0000 7fc0f3366700 1 -- 172.21.15.149:0/756134441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec0f5410 msgr2=0x7fc0ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:56.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.299+0000 7fc0f3366700 1 --2- 172.21.15.149:0/756134441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec0f5410 0x7fc0ec0f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fc0e0009fb0 tx=0x7fc0e0009b30).stop 2024-02-12T16:39:56.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.300+0000 7fc0f3366700 1 -- 172.21.15.149:0/756134441 shutdown_connections 2024-02-12T16:39:56.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.300+0000 7fc0f3366700 1 --2- 172.21.15.149:0/756134441 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec0f5410 0x7fc0ec0f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:56.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.300+0000 7fc0f3366700 1 -- 172.21.15.149:0/756134441 >> 172.21.15.149:0/756134441 conn(0x7fc0ec0f0b70 msgr2=0x7fc0ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:56.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.300+0000 7fc0f3366700 1 -- 172.21.15.149:0/756134441 shutdown_connections 2024-02-12T16:39:56.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.300+0000 7fc0f3366700 1 -- 172.21.15.149:0/756134441 wait complete. 2024-02-12T16:39:56.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.300+0000 7fc0f3366700 1 Processor -- start 2024-02-12T16:39:56.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.300+0000 7fc0f3366700 1 -- start start 2024-02-12T16:39:56.305 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.300+0000 7fc0f3366700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec0f5410 0x7fc0ec10b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.300+0000 7fc0f3366700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec107330 con 0x7fc0ec0f5410 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.300+0000 7fc0f1102700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec0f5410 0x7fc0ec10b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.301+0000 7fc0f1102700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec0f5410 0x7fc0ec10b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:36332/0 (socket says 172.21.15.149:36332) 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.301+0000 7fc0f1102700 1 -- 172.21.15.149:0/866199780 learned_addr learned my addr 172.21.15.149:0/866199780 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.301+0000 7fc0f1102700 1 -- 172.21.15.149:0/866199780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e0005040 con 0x7fc0ec0f5410 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.301+0000 7fc0f1102700 1 --2- 172.21.15.149:0/866199780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec0f5410 0x7fc0ec10b940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fc0e0005010 tx=0x7fc0e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.301+0000 7fc0ddffb700 1 -- 172.21.15.149:0/866199780 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0e0009320 con 0x7fc0ec0f5410 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.301+0000 7fc0ddffb700 1 -- 172.21.15.149:0/866199780 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0e0013380 con 0x7fc0ec0f5410 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.301+0000 7fc0f3366700 1 -- 172.21.15.149:0/866199780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ec060220 con 0x7fc0ec0f5410 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.301+0000 7fc0ddffb700 1 -- 172.21.15.149:0/866199780 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0e0014e50 con 0x7fc0ec0f5410 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.301+0000 7fc0f3366700 1 -- 172.21.15.149:0/866199780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ec10be30 con 0x7fc0ec0f5410 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.302+0000 7fc0ddffb700 1 -- 172.21.15.149:0/866199780 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0e00095f0 con 0x7fc0ec0f5410 2024-02-12T16:39:56.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.302+0000 7fc0f3366700 1 -- 172.21.15.149:0/866199780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0ec045ad0 con 0x7fc0ec0f5410 2024-02-12T16:39:56.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.302+0000 7fc0ddffb700 1 --2- 172.21.15.149:0/866199780 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc0d802ff10 0x7fc0d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:56.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.302+0000 7fc0ddffb700 1 -- 172.21.15.149:0/866199780 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0e0040e90 con 0x7fc0ec0f5410 2024-02-12T16:39:56.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.303+0000 7fc0f0901700 1 --2- 172.21.15.149:0/866199780 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc0d802ff10 0x7fc0d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:56.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.303+0000 7fc0f0901700 1 --2- 172.21.15.149:0/866199780 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc0d802ff10 0x7fc0d80323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fc0e80096a0 tx=0x7fc0e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:56.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.307+0000 7fc0ddffb700 1 -- 172.21.15.149:0/866199780 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0e0014420 con 0x7fc0ec0f5410 2024-02-12T16:39:56.513 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.511+0000 7fc0f3366700 1 -- 172.21.15.149:0/866199780 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0ec10c650 con 0x7fc0ec0f5410 2024-02-12T16:39:56.514 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.512+0000 7fc0ddffb700 1 -- 172.21.15.149:0/866199780 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc0e0010230 con 0x7fc0ec0f5410 2024-02-12T16:39:56.514 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:39:56.514 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:39:56.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.514+0000 7fc0f3366700 1 -- 172.21.15.149:0/866199780 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc0d802ff10 msgr2=0x7fc0d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:56.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.514+0000 7fc0f3366700 1 --2- 172.21.15.149:0/866199780 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc0d802ff10 0x7fc0d80323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fc0e80096a0 tx=0x7fc0e8006b40).stop 2024-02-12T16:39:56.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.514+0000 7fc0f3366700 1 -- 172.21.15.149:0/866199780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec0f5410 msgr2=0x7fc0ec10b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:56.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.514+0000 7fc0f3366700 1 --2- 172.21.15.149:0/866199780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec0f5410 0x7fc0ec10b940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fc0e0005010 tx=0x7fc0e000b040).stop 2024-02-12T16:39:56.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.514+0000 7fc0f3366700 1 -- 172.21.15.149:0/866199780 shutdown_connections 2024-02-12T16:39:56.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.514+0000 7fc0f3366700 1 --2- 172.21.15.149:0/866199780 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fc0d802ff10 0x7fc0d80323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:56.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.514+0000 7fc0f3366700 1 --2- 172.21.15.149:0/866199780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0ec0f5410 0x7fc0ec10b940 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:56.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.514+0000 7fc0f3366700 1 -- 172.21.15.149:0/866199780 >> 172.21.15.149:0/866199780 conn(0x7fc0ec0f0b70 msgr2=0x7fc0ec0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:56.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.514+0000 7fc0f3366700 1 -- 172.21.15.149:0/866199780 shutdown_connections 2024-02-12T16:39:56.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:56.514+0000 7fc0f3366700 1 -- 172.21.15.149:0/866199780 wait complete. 2024-02-12T16:39:56.517 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:39:56.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:56 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:54.372649+0000 mgr.smithi104.ohwbhf (mgr.14162) 323 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:57.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:57 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:39:56.513249+0000 mon.smithi104 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.149:0/866199780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:39:58.202 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:39:58.203 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:39:58.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:39:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:56.372958+0000 mgr.smithi104.ohwbhf (mgr.14162) 324 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:39:59.929 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.925+0000 7fca7f59e700 1 -- 172.21.15.149:0/3269531131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca800f51a0 msgr2=0x7fca800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:39:59.929 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.925+0000 7fca7f59e700 1 --2- 172.21.15.149:0/3269531131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca800f51a0 0x7fca800f55c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fca68009fb0 tx=0x7fca68009b30).stop 2024-02-12T16:39:59.929 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.925+0000 7fca7f59e700 1 -- 172.21.15.149:0/3269531131 shutdown_connections 2024-02-12T16:39:59.929 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.925+0000 7fca7f59e700 1 --2- 172.21.15.149:0/3269531131 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca800f51a0 0x7fca800f55c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.925+0000 7fca7f59e700 1 -- 172.21.15.149:0/3269531131 >> 172.21.15.149:0/3269531131 conn(0x7fca800f09a0 msgr2=0x7fca800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.925+0000 7fca7f59e700 1 -- 172.21.15.149:0/3269531131 shutdown_connections 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.925+0000 7fca7f59e700 1 -- 172.21.15.149:0/3269531131 wait complete. 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.926+0000 7fca7f59e700 1 Processor -- start 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.926+0000 7fca7f59e700 1 -- start start 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.926+0000 7fca7f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca800f51a0 0x7fca801054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.926+0000 7fca7f59e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca801059c0 con 0x7fca800f51a0 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.926+0000 7fca7e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca800f51a0 0x7fca801054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.926+0000 7fca7e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca800f51a0 0x7fca801054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:36344/0 (socket says 172.21.15.149:36344) 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.926+0000 7fca7e59c700 1 -- 172.21.15.149:0/2431626782 learned_addr learned my addr 172.21.15.149:0/2431626782 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.927+0000 7fca7e59c700 1 -- 172.21.15.149:0/2431626782 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca68005040 con 0x7fca800f51a0 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.927+0000 7fca7e59c700 1 --2- 172.21.15.149:0/2431626782 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca800f51a0 0x7fca801054d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fca68009830 tx=0x7fca6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:59.930 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.927+0000 7fca76ffd700 1 -- 172.21.15.149:0/2431626782 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca68009320 con 0x7fca800f51a0 2024-02-12T16:39:59.931 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.927+0000 7fca76ffd700 1 -- 172.21.15.149:0/2431626782 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca68013380 con 0x7fca800f51a0 2024-02-12T16:39:59.931 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.927+0000 7fca7f59e700 1 -- 172.21.15.149:0/2431626782 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca80105bc0 con 0x7fca800f51a0 2024-02-12T16:39:59.931 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.927+0000 7fca76ffd700 1 -- 172.21.15.149:0/2431626782 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca68014420 con 0x7fca800f51a0 2024-02-12T16:39:59.931 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.927+0000 7fca7f59e700 1 -- 172.21.15.149:0/2431626782 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca80106050 con 0x7fca800f51a0 2024-02-12T16:39:59.932 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.928+0000 7fca76ffd700 1 -- 172.21.15.149:0/2431626782 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca6801e420 con 0x7fca800f51a0 2024-02-12T16:39:59.932 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.928+0000 7fca7f59e700 1 -- 172.21.15.149:0/2431626782 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca80045ad0 con 0x7fca800f51a0 2024-02-12T16:39:59.932 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.928+0000 7fca76ffd700 1 --2- 172.21.15.149:0/2431626782 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fca6c02ffb0 0x7fca6c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:39:59.932 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.928+0000 7fca76ffd700 1 -- 172.21.15.149:0/2431626782 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca68040df0 con 0x7fca800f51a0 2024-02-12T16:39:59.932 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.928+0000 7fca7dd9b700 1 --2- 172.21.15.149:0/2431626782 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fca6c02ffb0 0x7fca6c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:39:59.932 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.929+0000 7fca7dd9b700 1 --2- 172.21.15.149:0/2431626782 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fca6c02ffb0 0x7fca6c032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fca700096a0 tx=0x7fca70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:39:59.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:39:59.933+0000 7fca76ffd700 1 -- 172.21.15.149:0/2431626782 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca6801cb10 con 0x7fca800f51a0 2024-02-12T16:40:00.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.136+0000 7fca7f59e700 1 -- 172.21.15.149:0/2431626782 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca800fad30 con 0x7fca800f51a0 2024-02-12T16:40:00.139 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.137+0000 7fca76ffd700 1 -- 172.21.15.149:0/2431626782 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fca6801cb10 con 0x7fca800f51a0 2024-02-12T16:40:00.139 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:00.140 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:00.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.139+0000 7fca7f59e700 1 -- 172.21.15.149:0/2431626782 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fca6c02ffb0 msgr2=0x7fca6c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:00.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.139+0000 7fca7f59e700 1 --2- 172.21.15.149:0/2431626782 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fca6c02ffb0 0x7fca6c032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fca700096a0 tx=0x7fca70006b40).stop 2024-02-12T16:40:00.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.139+0000 7fca7f59e700 1 -- 172.21.15.149:0/2431626782 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca800f51a0 msgr2=0x7fca801054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:00.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.139+0000 7fca7f59e700 1 --2- 172.21.15.149:0/2431626782 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca800f51a0 0x7fca801054d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fca68009830 tx=0x7fca6800b040).stop 2024-02-12T16:40:00.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.139+0000 7fca7f59e700 1 -- 172.21.15.149:0/2431626782 shutdown_connections 2024-02-12T16:40:00.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.139+0000 7fca7f59e700 1 --2- 172.21.15.149:0/2431626782 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fca6c02ffb0 0x7fca6c032470 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:00.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.139+0000 7fca7f59e700 1 --2- 172.21.15.149:0/2431626782 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca800f51a0 0x7fca801054d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:00.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.139+0000 7fca7f59e700 1 -- 172.21.15.149:0/2431626782 >> 172.21.15.149:0/2431626782 conn(0x7fca800f09a0 msgr2=0x7fca800f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:00.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.139+0000 7fca7f59e700 1 -- 172.21.15.149:0/2431626782 shutdown_connections 2024-02-12T16:40:00.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:00.139+0000 7fca7f59e700 1 -- 172.21.15.149:0/2431626782 wait complete. 2024-02-12T16:40:00.142 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:00.495 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:39:58.373272+0000 mgr.smithi104.ohwbhf (mgr.14162) 325 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:00.495 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:00 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:00.138531+0000 mon.smithi104 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.149:0/2431626782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:01.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:00.495223+0000 mon.smithi104 (mon.0) 308 : audit [DBG] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:40:01.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:00.721398+0000 mon.smithi104 (mon.0) 309 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:40:01.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:00.739088+0000 mon.smithi104 (mon.0) 310 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:40:01.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:00.971490+0000 mon.smithi104 (mon.0) 311 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:40:01.844 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:01.844 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:02.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:00.373562+0000 mgr.smithi104.ohwbhf (mgr.14162) 326 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:02.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:40:00.494581+0000 mgr.smithi104.ohwbhf (mgr.14162) 327 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi104 ... 2024-02-12T16:40:02.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-12T16:40:02.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-12T16:40:02.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:38:07Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:40:02.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:38:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:40:02.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:39:22Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:40:02.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-12T16:40:02.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Traceback (most recent call last): 2024-02-12T16:40:02.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: main() 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: r = ctx.func(ctx) 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: return func(ctx) 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: raise RuntimeError('uid/gid not found') 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: RuntimeError: uid/gid not found 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Traceback (most recent call last): 2024-02-12T16:40:02.427 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-12T16:40:02.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: yield (conn, connr) 2024-02-12T16:40:02.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-12T16:40:02.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: code, '\n'.join(err))) 2024-02-12T16:40:02.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi104 ... 2024-02-12T16:40:02.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi104 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-12T16:40:02.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-12T16:40:02.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:38:07Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:40:02.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:38:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:40:02.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr time="2024-02-12T16:39:22Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-12T16:40:02.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-12T16:40:02.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: Traceback (most recent call last): 2024-02-12T16:40:02.432 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: main() 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: r = ctx.func(ctx) 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: return func(ctx) 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: File "/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: raise RuntimeError('uid/gid not found') 2024-02-12T16:40:02.433 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: RuntimeError: uid/gid not found 2024-02-12T16:40:04.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:03 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:02.373836+0000 mgr.smithi104.ohwbhf (mgr.14162) 328 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:04.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:03 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:02.799407+0000 mon.smithi104 (mon.0) 312 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:40:04.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.880+0000 7fa4e7f64700 1 -- 172.21.15.149:0/2475522480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4e0060220 msgr2=0x7fa4e00fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:04.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.880+0000 7fa4e7f64700 1 --2- 172.21.15.149:0/2475522480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4e0060220 0x7fa4e00fb550 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fa4dc009fb0 tx=0x7fa4dc009b30).stop 2024-02-12T16:40:04.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.880+0000 7fa4e7f64700 1 -- 172.21.15.149:0/2475522480 shutdown_connections 2024-02-12T16:40:04.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.880+0000 7fa4e7f64700 1 --2- 172.21.15.149:0/2475522480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4e0060220 0x7fa4e00fb550 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:04.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.880+0000 7fa4e7f64700 1 -- 172.21.15.149:0/2475522480 >> 172.21.15.149:0/2475522480 conn(0x7fa4e00f0b70 msgr2=0x7fa4e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:04.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.880+0000 7fa4e7f64700 1 -- 172.21.15.149:0/2475522480 shutdown_connections 2024-02-12T16:40:04.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.881+0000 7fa4e7f64700 1 -- 172.21.15.149:0/2475522480 wait complete. 2024-02-12T16:40:04.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.881+0000 7fa4e7f64700 1 Processor -- start 2024-02-12T16:40:04.884 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.881+0000 7fa4e7f64700 1 -- start start 2024-02-12T16:40:04.885 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.881+0000 7fa4e7f64700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4e0060220 0x7fa4e01089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:04.885 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.881+0000 7fa4e7f64700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e0108e90 con 0x7fa4e0060220 2024-02-12T16:40:04.885 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.881+0000 7fa4e5d00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4e0060220 0x7fa4e01089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:04.885 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.881+0000 7fa4e5d00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4e0060220 0x7fa4e01089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:48966/0 (socket says 172.21.15.149:48966) 2024-02-12T16:40:04.885 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.881+0000 7fa4e5d00700 1 -- 172.21.15.149:0/3366858724 learned_addr learned my addr 172.21.15.149:0/3366858724 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:04.885 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.882+0000 7fa4e5d00700 1 -- 172.21.15.149:0/3366858724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc005040 con 0x7fa4e0060220 2024-02-12T16:40:04.885 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.882+0000 7fa4e5d00700 1 --2- 172.21.15.149:0/3366858724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4e0060220 0x7fa4e01089a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fa4dc0089f0 tx=0x7fa4dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:04.886 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.882+0000 7fa4d67fc700 1 -- 172.21.15.149:0/3366858724 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4dc009320 con 0x7fa4e0060220 2024-02-12T16:40:04.886 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.882+0000 7fa4e7f64700 1 -- 172.21.15.149:0/3366858724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e0109090 con 0x7fa4e0060220 2024-02-12T16:40:04.886 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.882+0000 7fa4d67fc700 1 -- 172.21.15.149:0/3366858724 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4dc013380 con 0x7fa4e0060220 2024-02-12T16:40:04.886 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.882+0000 7fa4d67fc700 1 -- 172.21.15.149:0/3366858724 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4dc014e20 con 0x7fa4e0060220 2024-02-12T16:40:04.886 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.882+0000 7fa4e7f64700 1 -- 172.21.15.149:0/3366858724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e0109520 con 0x7fa4e0060220 2024-02-12T16:40:04.886 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.883+0000 7fa4d67fc700 1 -- 172.21.15.149:0/3366858724 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4dc0095f0 con 0x7fa4e0060220 2024-02-12T16:40:04.887 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.883+0000 7fa4e7f64700 1 -- 172.21.15.149:0/3366858724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e0045ad0 con 0x7fa4e0060220 2024-02-12T16:40:04.887 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.883+0000 7fa4d67fc700 1 --2- 172.21.15.149:0/3366858724 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa4cc02ff10 0x7fa4cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:04.887 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.883+0000 7fa4d67fc700 1 -- 172.21.15.149:0/3366858724 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4dc040f50 con 0x7fa4e0060220 2024-02-12T16:40:04.887 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.883+0000 7fa4e54ff700 1 --2- 172.21.15.149:0/3366858724 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa4cc02ff10 0x7fa4cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:04.887 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.884+0000 7fa4e54ff700 1 --2- 172.21.15.149:0/3366858724 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa4cc02ff10 0x7fa4cc0323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fa4d00096a0 tx=0x7fa4d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:04.890 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:04.888+0000 7fa4d67fc700 1 -- 172.21.15.149:0/3366858724 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4dc014420 con 0x7fa4e0060220 2024-02-12T16:40:05.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.095+0000 7fa4e7f64700 1 -- 172.21.15.149:0/3366858724 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4e010a3d0 con 0x7fa4e0060220 2024-02-12T16:40:05.098 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.096+0000 7fa4d67fc700 1 -- 172.21.15.149:0/3366858724 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa4dc01d9d0 con 0x7fa4e0060220 2024-02-12T16:40:05.098 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:05.098 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:05.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.098+0000 7fa4e7f64700 1 -- 172.21.15.149:0/3366858724 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa4cc02ff10 msgr2=0x7fa4cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:05.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.098+0000 7fa4e7f64700 1 --2- 172.21.15.149:0/3366858724 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa4cc02ff10 0x7fa4cc0323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fa4d00096a0 tx=0x7fa4d0006b40).stop 2024-02-12T16:40:05.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.098+0000 7fa4e7f64700 1 -- 172.21.15.149:0/3366858724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4e0060220 msgr2=0x7fa4e01089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:05.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.098+0000 7fa4e7f64700 1 --2- 172.21.15.149:0/3366858724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4e0060220 0x7fa4e01089a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fa4dc0089f0 tx=0x7fa4dc00b040).stop 2024-02-12T16:40:05.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.098+0000 7fa4e7f64700 1 -- 172.21.15.149:0/3366858724 shutdown_connections 2024-02-12T16:40:05.100 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.098+0000 7fa4e7f64700 1 --2- 172.21.15.149:0/3366858724 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa4cc02ff10 0x7fa4cc0323d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:05.101 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.098+0000 7fa4e7f64700 1 --2- 172.21.15.149:0/3366858724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4e0060220 0x7fa4e01089a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:05.101 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.098+0000 7fa4e7f64700 1 -- 172.21.15.149:0/3366858724 >> 172.21.15.149:0/3366858724 conn(0x7fa4e00f0b70 msgr2=0x7fa4e00f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:05.101 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.098+0000 7fa4e7f64700 1 -- 172.21.15.149:0/3366858724 shutdown_connections 2024-02-12T16:40:05.101 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:05.098+0000 7fa4e7f64700 1 -- 172.21.15.149:0/3366858724 wait complete. 2024-02-12T16:40:05.101 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:06.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:04.374093+0000 mgr.smithi104.ohwbhf (mgr.14162) 329 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:06.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:04.689790+0000 mon.smithi104 (mon.0) 313 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:40:06.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:05.097427+0000 mon.smithi104 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.149:0/3366858724' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:06.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:05.519611+0000 mon.smithi104 (mon.0) 315 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:40:06.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:05.522814+0000 mon.smithi104 (mon.0) 316 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' 2024-02-12T16:40:06.692 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:06.692 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:07.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:05.520532+0000 mgr.smithi104.ohwbhf (mgr.14162) 330 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:07.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cephadm 2024-02-12T16:40:05.524915+0000 mgr.smithi104.ohwbhf (mgr.14162) 331 : cephadm [INF] Deploying daemon alertmanager.smithi104 on smithi104 2024-02-12T16:40:07.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:06.517944+0000 mon.smithi104 (mon.0) 317 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-12T16:40:08.559 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.553+0000 7f39285bc700 1 -- 172.21.15.149:0/135073784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39200f5410 msgr2=0x7f39200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:08.559 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.553+0000 7f39285bc700 1 --2- 172.21.15.149:0/135073784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39200f5410 0x7f39200f5830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f3910009fb0 tx=0x7f3910009b30).stop 2024-02-12T16:40:08.559 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.554+0000 7f39285bc700 1 -- 172.21.15.149:0/135073784 shutdown_connections 2024-02-12T16:40:08.560 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.554+0000 7f39285bc700 1 --2- 172.21.15.149:0/135073784 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39200f5410 0x7f39200f5830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:08.560 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.554+0000 7f39285bc700 1 -- 172.21.15.149:0/135073784 >> 172.21.15.149:0/135073784 conn(0x7f39200f0b70 msgr2=0x7f39200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:08.560 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.554+0000 7f39285bc700 1 -- 172.21.15.149:0/135073784 shutdown_connections 2024-02-12T16:40:08.560 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.554+0000 7f39285bc700 1 -- 172.21.15.149:0/135073784 wait complete. 2024-02-12T16:40:08.560 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.554+0000 7f39285bc700 1 Processor -- start 2024-02-12T16:40:08.560 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.554+0000 7f39285bc700 1 -- start start 2024-02-12T16:40:08.561 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.554+0000 7f39285bc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39200f5410 0x7f39200fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:08.561 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.554+0000 7f39285bc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39200fb960 con 0x7f39200f5410 2024-02-12T16:40:08.561 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.555+0000 7f3926358700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39200f5410 0x7f39200fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:08.561 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.555+0000 7f3926358700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39200f5410 0x7f39200fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:48986/0 (socket says 172.21.15.149:48986) 2024-02-12T16:40:08.561 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.555+0000 7f3926358700 1 -- 172.21.15.149:0/100181616 learned_addr learned my addr 172.21.15.149:0/100181616 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:08.561 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.555+0000 7f3926358700 1 -- 172.21.15.149:0/100181616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3910005040 con 0x7f39200f5410 2024-02-12T16:40:08.561 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.555+0000 7f3926358700 1 --2- 172.21.15.149:0/100181616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39200f5410 0x7f39200fd4e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f39100089f0 tx=0x7f391000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:08.561 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.555+0000 7f3916ffd700 1 -- 172.21.15.149:0/100181616 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3910009320 con 0x7f39200f5410 2024-02-12T16:40:08.561 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.555+0000 7f39285bc700 1 -- 172.21.15.149:0/100181616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39200fbb00 con 0x7f39200f5410 2024-02-12T16:40:08.561 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.555+0000 7f3916ffd700 1 -- 172.21.15.149:0/100181616 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3910013380 con 0x7f39200f5410 2024-02-12T16:40:08.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.555+0000 7f3916ffd700 1 -- 172.21.15.149:0/100181616 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f391001ed80 con 0x7f39200f5410 2024-02-12T16:40:08.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.555+0000 7f39285bc700 1 -- 172.21.15.149:0/100181616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39200fbf90 con 0x7f39200f5410 2024-02-12T16:40:08.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.556+0000 7f3916ffd700 1 -- 172.21.15.149:0/100181616 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f391001e420 con 0x7f39200f5410 2024-02-12T16:40:08.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.556+0000 7f39285bc700 1 -- 172.21.15.149:0/100181616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3920059100 con 0x7f39200f5410 2024-02-12T16:40:08.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.556+0000 7f3916ffd700 1 --2- 172.21.15.149:0/100181616 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f390c02ff60 0x7f390c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:08.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.556+0000 7f3916ffd700 1 -- 172.21.15.149:0/100181616 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3910041110 con 0x7f39200f5410 2024-02-12T16:40:08.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.556+0000 7f3925b57700 1 --2- 172.21.15.149:0/100181616 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f390c02ff60 0x7f390c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:08.562 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.557+0000 7f3925b57700 1 --2- 172.21.15.149:0/100181616 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f390c02ff60 0x7f390c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f391c0096a0 tx=0x7f391c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:08.563 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.561+0000 7f3916ffd700 1 -- 172.21.15.149:0/100181616 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f391001cb10 con 0x7f39200f5410 2024-02-12T16:40:08.766 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.764+0000 7f39285bc700 1 -- 172.21.15.149:0/100181616 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39200fafa0 con 0x7f39200f5410 2024-02-12T16:40:08.767 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.765+0000 7f3916ffd700 1 -- 172.21.15.149:0/100181616 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3910004490 con 0x7f39200f5410 2024-02-12T16:40:08.767 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:08.767 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:08.769 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.766+0000 7f39285bc700 1 -- 172.21.15.149:0/100181616 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f390c02ff60 msgr2=0x7f390c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:08.769 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.766+0000 7f39285bc700 1 --2- 172.21.15.149:0/100181616 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f390c02ff60 0x7f390c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f391c0096a0 tx=0x7f391c006b40).stop 2024-02-12T16:40:08.769 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.766+0000 7f39285bc700 1 -- 172.21.15.149:0/100181616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39200f5410 msgr2=0x7f39200fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:08.769 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.766+0000 7f39285bc700 1 --2- 172.21.15.149:0/100181616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39200f5410 0x7f39200fd4e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f39100089f0 tx=0x7f391000b040).stop 2024-02-12T16:40:08.769 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.766+0000 7f39285bc700 1 -- 172.21.15.149:0/100181616 shutdown_connections 2024-02-12T16:40:08.769 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.766+0000 7f39285bc700 1 --2- 172.21.15.149:0/100181616 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f390c02ff60 0x7f390c032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:08.769 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.766+0000 7f39285bc700 1 --2- 172.21.15.149:0/100181616 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39200f5410 0x7f39200fd4e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:08.769 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.766+0000 7f39285bc700 1 -- 172.21.15.149:0/100181616 >> 172.21.15.149:0/100181616 conn(0x7f39200f0b70 msgr2=0x7f392005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:08.769 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.767+0000 7f39285bc700 1 -- 172.21.15.149:0/100181616 shutdown_connections 2024-02-12T16:40:08.769 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:08.767+0000 7f39285bc700 1 -- 172.21.15.149:0/100181616 wait complete. 2024-02-12T16:40:08.770 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:09.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:07.520805+0000 mgr.smithi104.ohwbhf (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:09.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:08.766072+0000 mon.smithi104 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.149:0/100181616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:10.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:10.635 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:11.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:09.521097+0000 mgr.smithi104.ohwbhf (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:12.544 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.542+0000 7f253d142700 1 -- 172.21.15.149:0/543649502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25380f62a0 msgr2=0x7f25380f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:12.544 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.542+0000 7f253d142700 1 --2- 172.21.15.149:0/543649502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25380f62a0 0x7f25380f66c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f2528009fb0 tx=0x7f2528009b30).stop 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.542+0000 7f253d142700 1 -- 172.21.15.149:0/543649502 shutdown_connections 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.542+0000 7f253d142700 1 --2- 172.21.15.149:0/543649502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25380f62a0 0x7f25380f66c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.542+0000 7f253d142700 1 -- 172.21.15.149:0/543649502 >> 172.21.15.149:0/543649502 conn(0x7f25380f1aa0 msgr2=0x7f25380f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.542+0000 7f253d142700 1 -- 172.21.15.149:0/543649502 shutdown_connections 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.542+0000 7f253d142700 1 -- 172.21.15.149:0/543649502 wait complete. 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.542+0000 7f253d142700 1 Processor -- start 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.542+0000 7f253d142700 1 -- start start 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f253d142700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25380f62a0 0x7f253806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f253d142700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2538069e20 con 0x7f25380f62a0 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f2537fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25380f62a0 0x7f253806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f2537fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25380f62a0 0x7f253806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:49004/0 (socket says 172.21.15.149:49004) 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f2537fff700 1 -- 172.21.15.149:0/3310141716 learned_addr learned my addr 172.21.15.149:0/3310141716 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f2537fff700 1 -- 172.21.15.149:0/3310141716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2528005040 con 0x7f25380f62a0 2024-02-12T16:40:12.545 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f2537fff700 1 --2- 172.21.15.149:0/3310141716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25380f62a0 0x7f253806b9a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f25280089f0 tx=0x7f252800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:12.546 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f2534ff9700 1 -- 172.21.15.149:0/3310141716 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2528009320 con 0x7f25380f62a0 2024-02-12T16:40:12.546 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f253d142700 1 -- 172.21.15.149:0/3310141716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2538069fc0 con 0x7f25380f62a0 2024-02-12T16:40:12.546 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f2534ff9700 1 -- 172.21.15.149:0/3310141716 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2528013380 con 0x7f25380f62a0 2024-02-12T16:40:12.546 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f2534ff9700 1 -- 172.21.15.149:0/3310141716 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2528014e20 con 0x7f25380f62a0 2024-02-12T16:40:12.546 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.543+0000 7f253d142700 1 -- 172.21.15.149:0/3310141716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253806a450 con 0x7f25380f62a0 2024-02-12T16:40:12.547 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.544+0000 7f2534ff9700 1 -- 172.21.15.149:0/3310141716 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25280263a0 con 0x7f25380f62a0 2024-02-12T16:40:12.547 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.544+0000 7f2534ff9700 1 --2- 172.21.15.149:0/3310141716 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f252002ff60 0x7f2520032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:12.547 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.544+0000 7f2534ff9700 1 -- 172.21.15.149:0/3310141716 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2528040f20 con 0x7f25380f62a0 2024-02-12T16:40:12.547 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.544+0000 7f253d142700 1 -- 172.21.15.149:0/3310141716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2538059100 con 0x7f25380f62a0 2024-02-12T16:40:12.547 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.545+0000 7f25377fe700 1 --2- 172.21.15.149:0/3310141716 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f252002ff60 0x7f2520032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:12.547 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.545+0000 7f25377fe700 1 --2- 172.21.15.149:0/3310141716 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f252002ff60 0x7f2520032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f252c0096a0 tx=0x7f252c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:12.550 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.549+0000 7f2534ff9700 1 -- 172.21.15.149:0/3310141716 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25280672b0 con 0x7f25380f62a0 2024-02-12T16:40:12.756 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.754+0000 7f253d142700 1 -- 172.21.15.149:0/3310141716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25380fbe30 con 0x7f25380f62a0 2024-02-12T16:40:12.756 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.755+0000 7f2534ff9700 1 -- 172.21.15.149:0/3310141716 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f252802b020 con 0x7f25380f62a0 2024-02-12T16:40:12.756 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:12.757 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:12.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.757+0000 7f253d142700 1 -- 172.21.15.149:0/3310141716 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f252002ff60 msgr2=0x7f2520032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:12.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.757+0000 7f253d142700 1 --2- 172.21.15.149:0/3310141716 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f252002ff60 0x7f2520032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f252c0096a0 tx=0x7f252c006b40).stop 2024-02-12T16:40:12.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.757+0000 7f253d142700 1 -- 172.21.15.149:0/3310141716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25380f62a0 msgr2=0x7f253806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:12.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.757+0000 7f253d142700 1 --2- 172.21.15.149:0/3310141716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25380f62a0 0x7f253806b9a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f25280089f0 tx=0x7f252800b040).stop 2024-02-12T16:40:12.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.757+0000 7f253d142700 1 -- 172.21.15.149:0/3310141716 shutdown_connections 2024-02-12T16:40:12.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.757+0000 7f253d142700 1 --2- 172.21.15.149:0/3310141716 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f252002ff60 0x7f2520032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:12.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.757+0000 7f253d142700 1 --2- 172.21.15.149:0/3310141716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25380f62a0 0x7f253806b9a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:12.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.757+0000 7f253d142700 1 -- 172.21.15.149:0/3310141716 >> 172.21.15.149:0/3310141716 conn(0x7f25380f1aa0 msgr2=0x7f25380f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:12.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.757+0000 7f253d142700 1 -- 172.21.15.149:0/3310141716 shutdown_connections 2024-02-12T16:40:12.759 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:12.757+0000 7f253d142700 1 -- 172.21.15.149:0/3310141716 wait complete. 2024-02-12T16:40:12.759 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:13.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:11.521355+0000 mgr.smithi104.ohwbhf (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:13.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:12.755523+0000 mon.smithi104 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.149:0/3310141716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:14.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:14.476 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:15.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:15 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:13.521651+0000 mgr.smithi104.ohwbhf (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:16.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.302+0000 7fab51bcc700 1 -- 172.21.15.149:0/3644204983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab4c0f5410 msgr2=0x7fab4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:16.303 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.302+0000 7fab51bcc700 1 --2- 172.21.15.149:0/3644204983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab4c0f5410 0x7fab4c0f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fab3c009fb0 tx=0x7fab3c009b30).stop 2024-02-12T16:40:16.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.302+0000 7fab51bcc700 1 -- 172.21.15.149:0/3644204983 shutdown_connections 2024-02-12T16:40:16.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.302+0000 7fab51bcc700 1 --2- 172.21.15.149:0/3644204983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab4c0f5410 0x7fab4c0f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:16.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.302+0000 7fab51bcc700 1 -- 172.21.15.149:0/3644204983 >> 172.21.15.149:0/3644204983 conn(0x7fab4c0f0b70 msgr2=0x7fab4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:16.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.302+0000 7fab51bcc700 1 -- 172.21.15.149:0/3644204983 shutdown_connections 2024-02-12T16:40:16.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.302+0000 7fab51bcc700 1 -- 172.21.15.149:0/3644204983 wait complete. 2024-02-12T16:40:16.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.302+0000 7fab51bcc700 1 Processor -- start 2024-02-12T16:40:16.304 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.302+0000 7fab51bcc700 1 -- start start 2024-02-12T16:40:16.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.303+0000 7fab51bcc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab4c0f5410 0x7fab4c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:16.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.303+0000 7fab51bcc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c105bd0 con 0x7fab4c0f5410 2024-02-12T16:40:16.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.303+0000 7fab4b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab4c0f5410 0x7fab4c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:16.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.303+0000 7fab4b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab4c0f5410 0x7fab4c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:45084/0 (socket says 172.21.15.149:45084) 2024-02-12T16:40:16.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.303+0000 7fab4b7fe700 1 -- 172.21.15.149:0/62059548 learned_addr learned my addr 172.21.15.149:0/62059548 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:16.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.303+0000 7fab4b7fe700 1 -- 172.21.15.149:0/62059548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c005040 con 0x7fab4c0f5410 2024-02-12T16:40:16.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.303+0000 7fab4b7fe700 1 --2- 172.21.15.149:0/62059548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab4c0f5410 0x7fab4c1056e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fab3c0089f0 tx=0x7fab3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:16.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.304+0000 7fab33fff700 1 -- 172.21.15.149:0/62059548 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab3c009320 con 0x7fab4c0f5410 2024-02-12T16:40:16.306 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.304+0000 7fab51bcc700 1 -- 172.21.15.149:0/62059548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4c105dd0 con 0x7fab4c0f5410 2024-02-12T16:40:16.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.304+0000 7fab33fff700 1 -- 172.21.15.149:0/62059548 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab3c013380 con 0x7fab4c0f5410 2024-02-12T16:40:16.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.304+0000 7fab33fff700 1 -- 172.21.15.149:0/62059548 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab3c01ed80 con 0x7fab4c0f5410 2024-02-12T16:40:16.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.304+0000 7fab51bcc700 1 -- 172.21.15.149:0/62059548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4c106260 con 0x7fab4c0f5410 2024-02-12T16:40:16.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.304+0000 7fab33fff700 1 -- 172.21.15.149:0/62059548 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab3c014420 con 0x7fab4c0f5410 2024-02-12T16:40:16.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.305+0000 7fab51bcc700 1 -- 172.21.15.149:0/62059548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4c045ad0 con 0x7fab4c0f5410 2024-02-12T16:40:16.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.305+0000 7fab33fff700 1 --2- 172.21.15.149:0/62059548 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab3402ff60 0x7fab34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:16.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.305+0000 7fab33fff700 1 -- 172.21.15.149:0/62059548 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab3c0410c0 con 0x7fab4c0f5410 2024-02-12T16:40:16.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.305+0000 7fab4affd700 1 --2- 172.21.15.149:0/62059548 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab3402ff60 0x7fab34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:16.308 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.305+0000 7fab4affd700 1 --2- 172.21.15.149:0/62059548 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab3402ff60 0x7fab34032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fab400096a0 tx=0x7fab40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:16.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.309+0000 7fab33fff700 1 -- 172.21.15.149:0/62059548 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab3c0095f0 con 0x7fab4c0f5410 2024-02-12T16:40:16.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.514+0000 7fab51bcc700 1 -- 172.21.15.149:0/62059548 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab4c107110 con 0x7fab4c0f5410 2024-02-12T16:40:16.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.515+0000 7fab33fff700 1 -- 172.21.15.149:0/62059548 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fab3c004490 con 0x7fab4c0f5410 2024-02-12T16:40:16.516 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:16.516 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:16.517 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.516+0000 7fab51bcc700 1 -- 172.21.15.149:0/62059548 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab3402ff60 msgr2=0x7fab34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:16.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.516+0000 7fab51bcc700 1 --2- 172.21.15.149:0/62059548 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab3402ff60 0x7fab34032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fab400096a0 tx=0x7fab40006b40).stop 2024-02-12T16:40:16.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.516+0000 7fab51bcc700 1 -- 172.21.15.149:0/62059548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab4c0f5410 msgr2=0x7fab4c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:16.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.516+0000 7fab51bcc700 1 --2- 172.21.15.149:0/62059548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab4c0f5410 0x7fab4c1056e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fab3c0089f0 tx=0x7fab3c00b040).stop 2024-02-12T16:40:16.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.516+0000 7fab51bcc700 1 -- 172.21.15.149:0/62059548 shutdown_connections 2024-02-12T16:40:16.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.516+0000 7fab51bcc700 1 --2- 172.21.15.149:0/62059548 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fab3402ff60 0x7fab34032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:16.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.516+0000 7fab51bcc700 1 --2- 172.21.15.149:0/62059548 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fab4c0f5410 0x7fab4c1056e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:16.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.516+0000 7fab51bcc700 1 -- 172.21.15.149:0/62059548 >> 172.21.15.149:0/62059548 conn(0x7fab4c0f0b70 msgr2=0x7fab4c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:16.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.517+0000 7fab51bcc700 1 -- 172.21.15.149:0/62059548 shutdown_connections 2024-02-12T16:40:16.518 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:16.517+0000 7fab51bcc700 1 -- 172.21.15.149:0/62059548 wait complete. 2024-02-12T16:40:16.519 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:17.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:17 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:15.522037+0000 mgr.smithi104.ohwbhf (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:17.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:17 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:16.515046+0000 mon.smithi104 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.149:0/62059548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:18.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:18.233 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:19.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:19 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:17.522415+0000 mgr.smithi104.ohwbhf (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:20.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.136+0000 7f99e7429700 1 -- 172.21.15.149:0/1033504501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99e00f6510 msgr2=0x7f99e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:20.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.136+0000 7f99e7429700 1 --2- 172.21.15.149:0/1033504501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99e00f6510 0x7f99e00f6930 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f99d0009fb0 tx=0x7f99d0009b30).stop 2024-02-12T16:40:20.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.136+0000 7f99e7429700 1 -- 172.21.15.149:0/1033504501 shutdown_connections 2024-02-12T16:40:20.141 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.136+0000 7f99e7429700 1 --2- 172.21.15.149:0/1033504501 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99e00f6510 0x7f99e00f6930 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:20.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.136+0000 7f99e7429700 1 -- 172.21.15.149:0/1033504501 >> 172.21.15.149:0/1033504501 conn(0x7f99e00f1c70 msgr2=0x7f99e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:20.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.137+0000 7f99e7429700 1 -- 172.21.15.149:0/1033504501 shutdown_connections 2024-02-12T16:40:20.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.137+0000 7f99e7429700 1 -- 172.21.15.149:0/1033504501 wait complete. 2024-02-12T16:40:20.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.137+0000 7f99e7429700 1 Processor -- start 2024-02-12T16:40:20.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.137+0000 7f99e7429700 1 -- start start 2024-02-12T16:40:20.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.137+0000 7f99e7429700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99e006b960 0x7f99e0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:20.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.137+0000 7f99e7429700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e006bd80 con 0x7f99e006b960 2024-02-12T16:40:20.142 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.137+0000 7f99e51c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99e006b960 0x7f99e0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:20.143 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.137+0000 7f99e51c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99e006b960 0x7f99e0069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:45090/0 (socket says 172.21.15.149:45090) 2024-02-12T16:40:20.143 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.137+0000 7f99e51c5700 1 -- 172.21.15.149:0/2006802238 learned_addr learned my addr 172.21.15.149:0/2006802238 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:20.143 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.138+0000 7f99e51c5700 1 -- 172.21.15.149:0/2006802238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d0005040 con 0x7f99e006b960 2024-02-12T16:40:20.143 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.138+0000 7f99e51c5700 1 --2- 172.21.15.149:0/2006802238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99e006b960 0x7f99e0069e20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f99e00f7610 tx=0x7f99d0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:20.143 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.138+0000 7f99d5ffb700 1 -- 172.21.15.149:0/2006802238 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d0013380 con 0x7f99e006b960 2024-02-12T16:40:20.143 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.138+0000 7f99e7429700 1 -- 172.21.15.149:0/2006802238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e006a310 con 0x7f99e006b960 2024-02-12T16:40:20.144 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.138+0000 7f99d5ffb700 1 -- 172.21.15.149:0/2006802238 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99d0013790 con 0x7f99e006b960 2024-02-12T16:40:20.144 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.138+0000 7f99d5ffb700 1 -- 172.21.15.149:0/2006802238 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d001ee70 con 0x7f99e006b960 2024-02-12T16:40:20.144 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.138+0000 7f99e7429700 1 -- 172.21.15.149:0/2006802238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e006a800 con 0x7f99e006b960 2024-02-12T16:40:20.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.139+0000 7f99e7429700 1 -- 172.21.15.149:0/2006802238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e0059100 con 0x7f99e006b960 2024-02-12T16:40:20.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.141+0000 7f99d5ffb700 1 -- 172.21.15.149:0/2006802238 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99d0025060 con 0x7f99e006b960 2024-02-12T16:40:20.145 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.141+0000 7f99d5ffb700 1 --2- 172.21.15.149:0/2006802238 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f99cc02ff60 0x7f99cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:20.146 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.141+0000 7f99d5ffb700 1 -- 172.21.15.149:0/2006802238 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99d0041160 con 0x7f99e006b960 2024-02-12T16:40:20.146 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.142+0000 7f99e49c4700 1 --2- 172.21.15.149:0/2006802238 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f99cc02ff60 0x7f99cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:20.146 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.142+0000 7f99e49c4700 1 --2- 172.21.15.149:0/2006802238 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f99cc02ff60 0x7f99cc032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f99dc0096a0 tx=0x7f99dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:20.147 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.145+0000 7f99d5ffb700 1 -- 172.21.15.149:0/2006802238 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99d001c950 con 0x7f99e006b960 2024-02-12T16:40:20.352 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.350+0000 7f99e7429700 1 -- 172.21.15.149:0/2006802238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99e0045ad0 con 0x7f99e006b960 2024-02-12T16:40:20.353 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.350+0000 7f99d5ffb700 1 -- 172.21.15.149:0/2006802238 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f99d001cb60 con 0x7f99e006b960 2024-02-12T16:40:20.353 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:20.353 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:20.354 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.352+0000 7f99e7429700 1 -- 172.21.15.149:0/2006802238 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f99cc02ff60 msgr2=0x7f99cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:20.354 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.352+0000 7f99e7429700 1 --2- 172.21.15.149:0/2006802238 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f99cc02ff60 0x7f99cc032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f99dc0096a0 tx=0x7f99dc006b40).stop 2024-02-12T16:40:20.355 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.352+0000 7f99e7429700 1 -- 172.21.15.149:0/2006802238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99e006b960 msgr2=0x7f99e0069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:20.355 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.352+0000 7f99e7429700 1 --2- 172.21.15.149:0/2006802238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99e006b960 0x7f99e0069e20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f99e00f7610 tx=0x7f99d0009b30).stop 2024-02-12T16:40:20.355 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.352+0000 7f99e7429700 1 -- 172.21.15.149:0/2006802238 shutdown_connections 2024-02-12T16:40:20.355 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.352+0000 7f99e7429700 1 --2- 172.21.15.149:0/2006802238 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f99cc02ff60 0x7f99cc032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:20.355 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.352+0000 7f99e7429700 1 --2- 172.21.15.149:0/2006802238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99e006b960 0x7f99e0069e20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:20.355 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.352+0000 7f99e7429700 1 -- 172.21.15.149:0/2006802238 >> 172.21.15.149:0/2006802238 conn(0x7f99e00f1c70 msgr2=0x7f99e00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:20.355 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.352+0000 7f99e7429700 1 -- 172.21.15.149:0/2006802238 shutdown_connections 2024-02-12T16:40:20.355 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:20.352+0000 7f99e7429700 1 -- 172.21.15.149:0/2006802238 wait complete. 2024-02-12T16:40:20.355 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:21.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:21 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:19.522815+0000 mgr.smithi104.ohwbhf (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:21.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:21 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:20.351700+0000 mon.smithi104 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.149:0/2006802238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:21.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:21.981 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:23.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:23 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:21.523134+0000 mgr.smithi104.ohwbhf (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:23.874 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.860+0000 7fa2fec34700 1 -- 172.21.15.149:0/2137023560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2f8060280 msgr2=0x7fa2f80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.860+0000 7fa2fec34700 1 --2- 172.21.15.149:0/2137023560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2f8060280 0x7fa2f80f90f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fa2ec009fb0 tx=0x7fa2ec009b30).stop 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.860+0000 7fa2fec34700 1 -- 172.21.15.149:0/2137023560 shutdown_connections 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.861+0000 7fa2fec34700 1 --2- 172.21.15.149:0/2137023560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2f8060280 0x7fa2f80f90f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.861+0000 7fa2fec34700 1 -- 172.21.15.149:0/2137023560 >> 172.21.15.149:0/2137023560 conn(0x7fa2f80f0b70 msgr2=0x7fa2f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.861+0000 7fa2fec34700 1 -- 172.21.15.149:0/2137023560 shutdown_connections 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.861+0000 7fa2fec34700 1 -- 172.21.15.149:0/2137023560 wait complete. 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.861+0000 7fa2fec34700 1 Processor -- start 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.861+0000 7fa2fec34700 1 -- start start 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.861+0000 7fa2fec34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2f8060280 0x7fa2f80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.861+0000 7fa2fec34700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80f4ff0 con 0x7fa2f8060280 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.862+0000 7fa2fc9d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2f8060280 0x7fa2f80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.862+0000 7fa2fc9d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2f8060280 0x7fa2f80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:45100/0 (socket says 172.21.15.149:45100) 2024-02-12T16:40:23.875 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.862+0000 7fa2fc9d0700 1 -- 172.21.15.149:0/77575068 learned_addr learned my addr 172.21.15.149:0/77575068 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.862+0000 7fa2fc9d0700 1 -- 172.21.15.149:0/77575068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec005040 con 0x7fa2f8060280 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.862+0000 7fa2fc9d0700 1 --2- 172.21.15.149:0/77575068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2f8060280 0x7fa2f80f6b70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fa2ec0089f0 tx=0x7fa2ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.862+0000 7fa2f57fa700 1 -- 172.21.15.149:0/77575068 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ec009320 con 0x7fa2f8060280 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.862+0000 7fa2fec34700 1 -- 172.21.15.149:0/77575068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f80f5190 con 0x7fa2f8060280 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.862+0000 7fa2f57fa700 1 -- 172.21.15.149:0/77575068 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2ec013380 con 0x7fa2f8060280 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.862+0000 7fa2f57fa700 1 -- 172.21.15.149:0/77575068 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ec014e20 con 0x7fa2f8060280 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.863+0000 7fa2fec34700 1 -- 172.21.15.149:0/77575068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f80f5620 con 0x7fa2f8060280 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.863+0000 7fa2f57fa700 1 -- 172.21.15.149:0/77575068 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2ec014420 con 0x7fa2f8060280 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.863+0000 7fa2fec34700 1 -- 172.21.15.149:0/77575068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f8059100 con 0x7fa2f8060280 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.863+0000 7fa2f57fa700 1 --2- 172.21.15.149:0/77575068 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa2e802ff10 0x7fa2e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.863+0000 7fa2f57fa700 1 -- 172.21.15.149:0/77575068 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2ec040f00 con 0x7fa2f8060280 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.864+0000 7fa2f7fff700 1 --2- 172.21.15.149:0/77575068 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa2e802ff10 0x7fa2e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.864+0000 7fa2f7fff700 1 --2- 172.21.15.149:0/77575068 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa2e802ff10 0x7fa2e80323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa2e40096a0 tx=0x7fa2e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:23.876 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:23.868+0000 7fa2f57fa700 1 -- 172.21.15.149:0/77575068 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2ec0095f0 con 0x7fa2f8060280 2024-02-12T16:40:24.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.076+0000 7fa2fec34700 1 -- 172.21.15.149:0/77575068 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2f80fc090 con 0x7fa2f8060280 2024-02-12T16:40:24.079 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.077+0000 7fa2f57fa700 1 -- 172.21.15.149:0/77575068 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa2ec01db70 con 0x7fa2f8060280 2024-02-12T16:40:24.080 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:24.080 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:24.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.079+0000 7fa2fec34700 1 -- 172.21.15.149:0/77575068 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa2e802ff10 msgr2=0x7fa2e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:24.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.079+0000 7fa2fec34700 1 --2- 172.21.15.149:0/77575068 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa2e802ff10 0x7fa2e80323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa2e40096a0 tx=0x7fa2e4006b40).stop 2024-02-12T16:40:24.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.079+0000 7fa2fec34700 1 -- 172.21.15.149:0/77575068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2f8060280 msgr2=0x7fa2f80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:24.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.079+0000 7fa2fec34700 1 --2- 172.21.15.149:0/77575068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2f8060280 0x7fa2f80f6b70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fa2ec0089f0 tx=0x7fa2ec00b040).stop 2024-02-12T16:40:24.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.079+0000 7fa2fec34700 1 -- 172.21.15.149:0/77575068 shutdown_connections 2024-02-12T16:40:24.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.079+0000 7fa2fec34700 1 --2- 172.21.15.149:0/77575068 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fa2e802ff10 0x7fa2e80323d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:24.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.079+0000 7fa2fec34700 1 --2- 172.21.15.149:0/77575068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa2f8060280 0x7fa2f80f6b70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:24.081 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.079+0000 7fa2fec34700 1 -- 172.21.15.149:0/77575068 >> 172.21.15.149:0/77575068 conn(0x7fa2f80f0b70 msgr2=0x7fa2f80f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:24.082 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.079+0000 7fa2fec34700 1 -- 172.21.15.149:0/77575068 shutdown_connections 2024-02-12T16:40:24.082 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:24.079+0000 7fa2fec34700 1 -- 172.21.15.149:0/77575068 wait complete. 2024-02-12T16:40:24.082 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:24.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:24.078548+0000 mon.smithi104 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.149:0/77575068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:23.523465+0000 mgr.smithi104.ohwbhf (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:25.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:24.414472+0000 mon.smithi104 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/mirror_snapshot_schedule"}]: dispatch 2024-02-12T16:40:25.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:25 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:24.419078+0000 mon.smithi104 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.104:0/529522373' entity='mgr.smithi104.ohwbhf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.ohwbhf/trash_purge_schedule"}]: dispatch 2024-02-12T16:40:25.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:25.713 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:27.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:27 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:25.523854+0000 mgr.smithi104.ohwbhf (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.503+0000 7f3966337700 1 -- 172.21.15.149:0/4132886151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39600f6510 msgr2=0x7f39600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.503+0000 7f3966337700 1 --2- 172.21.15.149:0/4132886151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39600f6510 0x7f39600f6930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f3950009fb0 tx=0x7f3950009b30).stop 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.503+0000 7f3966337700 1 -- 172.21.15.149:0/4132886151 shutdown_connections 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.503+0000 7f3966337700 1 --2- 172.21.15.149:0/4132886151 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39600f6510 0x7f39600f6930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.503+0000 7f3966337700 1 -- 172.21.15.149:0/4132886151 >> 172.21.15.149:0/4132886151 conn(0x7f39600f1c70 msgr2=0x7f39600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.503+0000 7f3966337700 1 -- 172.21.15.149:0/4132886151 shutdown_connections 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.503+0000 7f3966337700 1 -- 172.21.15.149:0/4132886151 wait complete. 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.504+0000 7f3966337700 1 Processor -- start 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.504+0000 7f3966337700 1 -- start start 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.504+0000 7f3966337700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39600f6510 0x7f396006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.504+0000 7f3966337700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3960069e20 con 0x7f39600f6510 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.504+0000 7f395ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39600f6510 0x7f396006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.504+0000 7f395ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39600f6510 0x7f396006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:42644/0 (socket says 172.21.15.149:42644) 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.504+0000 7f395ffff700 1 -- 172.21.15.149:0/1357339785 learned_addr learned my addr 172.21.15.149:0/1357339785 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.505+0000 7f395ffff700 1 -- 172.21.15.149:0/1357339785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3950005040 con 0x7f39600f6510 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.505+0000 7f395ffff700 1 --2- 172.21.15.149:0/1357339785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39600f6510 0x7f396006b9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f3950005010 tx=0x7f395000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.505+0000 7f395cff9700 1 -- 172.21.15.149:0/1357339785 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3950009320 con 0x7f39600f6510 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.505+0000 7f3966337700 1 -- 172.21.15.149:0/1357339785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3960069fc0 con 0x7f39600f6510 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.505+0000 7f395cff9700 1 -- 172.21.15.149:0/1357339785 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3950013380 con 0x7f39600f6510 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.505+0000 7f395cff9700 1 -- 172.21.15.149:0/1357339785 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3950014e50 con 0x7f39600f6510 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.505+0000 7f3966337700 1 -- 172.21.15.149:0/1357339785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396006a450 con 0x7f39600f6510 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.506+0000 7f395cff9700 1 -- 172.21.15.149:0/1357339785 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3950014420 con 0x7f39600f6510 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.506+0000 7f3966337700 1 -- 172.21.15.149:0/1357339785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3960059100 con 0x7f39600f6510 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.506+0000 7f395cff9700 1 --2- 172.21.15.149:0/1357339785 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f394802ff10 0x7f39480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.506+0000 7f395cff9700 1 -- 172.21.15.149:0/1357339785 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3950040f40 con 0x7f39600f6510 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.506+0000 7f395f7fe700 1 --2- 172.21.15.149:0/1357339785 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f394802ff10 0x7f39480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.506+0000 7f395f7fe700 1 --2- 172.21.15.149:0/1357339785 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f394802ff10 0x7f39480323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f39540096a0 tx=0x7f3954006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.511+0000 7f395cff9700 1 -- 172.21.15.149:0/1357339785 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39500095f0 con 0x7f39600f6510 2024-02-12T16:40:27.717 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.715+0000 7f3966337700 1 -- 172.21.15.149:0/1357339785 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39600fc0a0 con 0x7f39600f6510 2024-02-12T16:40:27.718 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.716+0000 7f395cff9700 1 -- 172.21.15.149:0/1357339785 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f395001db70 con 0x7f39600f6510 2024-02-12T16:40:27.718 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:27.718 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:27.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.717+0000 7f3966337700 1 -- 172.21.15.149:0/1357339785 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f394802ff10 msgr2=0x7f39480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:27.720 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.717+0000 7f3966337700 1 --2- 172.21.15.149:0/1357339785 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f394802ff10 0x7f39480323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f39540096a0 tx=0x7f3954006b40).stop 2024-02-12T16:40:27.720 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.717+0000 7f3966337700 1 -- 172.21.15.149:0/1357339785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39600f6510 msgr2=0x7f396006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:27.720 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.717+0000 7f3966337700 1 --2- 172.21.15.149:0/1357339785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39600f6510 0x7f396006b9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f3950005010 tx=0x7f395000b040).stop 2024-02-12T16:40:27.720 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.717+0000 7f3966337700 1 -- 172.21.15.149:0/1357339785 shutdown_connections 2024-02-12T16:40:27.720 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.717+0000 7f3966337700 1 --2- 172.21.15.149:0/1357339785 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f394802ff10 0x7f39480323d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:27.720 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.717+0000 7f3966337700 1 --2- 172.21.15.149:0/1357339785 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39600f6510 0x7f396006b9a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:27.720 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.717+0000 7f3966337700 1 -- 172.21.15.149:0/1357339785 >> 172.21.15.149:0/1357339785 conn(0x7f39600f1c70 msgr2=0x7f39600f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:27.720 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.717+0000 7f3966337700 1 -- 172.21.15.149:0/1357339785 shutdown_connections 2024-02-12T16:40:27.720 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:27.717+0000 7f3966337700 1 -- 172.21.15.149:0/1357339785 wait complete. 2024-02-12T16:40:27.721 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:28.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:28 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:27.716864+0000 mon.smithi104 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.149:0/1357339785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:29.385 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:29.386 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:29.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:29 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:27.524266+0000 mgr.smithi104.ohwbhf (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:31.277 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.273+0000 7f68613c6700 1 -- 172.21.15.149:0/3788265208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685c0f5410 msgr2=0x7f685c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:31.277 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.273+0000 7f68613c6700 1 --2- 172.21.15.149:0/3788265208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685c0f5410 0x7f685c0f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6850009fb0 tx=0x7f6850009b30).stop 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.273+0000 7f68613c6700 1 -- 172.21.15.149:0/3788265208 shutdown_connections 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.273+0000 7f68613c6700 1 --2- 172.21.15.149:0/3788265208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685c0f5410 0x7f685c0f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.273+0000 7f68613c6700 1 -- 172.21.15.149:0/3788265208 >> 172.21.15.149:0/3788265208 conn(0x7f685c0f0b70 msgr2=0x7f685c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.273+0000 7f68613c6700 1 -- 172.21.15.149:0/3788265208 shutdown_connections 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.273+0000 7f68613c6700 1 -- 172.21.15.149:0/3788265208 wait complete. 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.274+0000 7f68613c6700 1 Processor -- start 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.274+0000 7f68613c6700 1 -- start start 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.274+0000 7f68613c6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685c0f5410 0x7f685c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.274+0000 7f68613c6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c106d00 con 0x7f685c0f5410 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.274+0000 7f685affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685c0f5410 0x7f685c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.274+0000 7f685affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685c0f5410 0x7f685c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:42662/0 (socket says 172.21.15.149:42662) 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.274+0000 7f685affd700 1 -- 172.21.15.149:0/4212051694 learned_addr learned my addr 172.21.15.149:0/4212051694 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:31.278 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.275+0000 7f685affd700 1 -- 172.21.15.149:0/4212051694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6850005040 con 0x7f685c0f5410 2024-02-12T16:40:31.279 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.275+0000 7f685affd700 1 --2- 172.21.15.149:0/4212051694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685c0f5410 0x7f685c106810 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f6850005010 tx=0x7f685000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:31.279 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.275+0000 7f68437fe700 1 -- 172.21.15.149:0/4212051694 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6850009320 con 0x7f685c0f5410 2024-02-12T16:40:31.279 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.275+0000 7f68437fe700 1 -- 172.21.15.149:0/4212051694 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6850013380 con 0x7f685c0f5410 2024-02-12T16:40:31.279 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.275+0000 7f68613c6700 1 -- 172.21.15.149:0/4212051694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685c106f00 con 0x7f685c0f5410 2024-02-12T16:40:31.279 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.275+0000 7f68437fe700 1 -- 172.21.15.149:0/4212051694 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6850014e50 con 0x7f685c0f5410 2024-02-12T16:40:31.279 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.275+0000 7f68613c6700 1 -- 172.21.15.149:0/4212051694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685c107390 con 0x7f685c0f5410 2024-02-12T16:40:31.280 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.276+0000 7f68437fe700 1 -- 172.21.15.149:0/4212051694 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68500095f0 con 0x7f685c0f5410 2024-02-12T16:40:31.280 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.276+0000 7f68613c6700 1 -- 172.21.15.149:0/4212051694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685c045ad0 con 0x7f685c0f5410 2024-02-12T16:40:31.280 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.276+0000 7f68437fe700 1 --2- 172.21.15.149:0/4212051694 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f684402ff10 0x7f68440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:31.280 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.276+0000 7f68437fe700 1 -- 172.21.15.149:0/4212051694 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6850040fa0 con 0x7f685c0f5410 2024-02-12T16:40:31.280 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.276+0000 7f685a7fc700 1 --2- 172.21.15.149:0/4212051694 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f684402ff10 0x7f68440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:31.280 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.277+0000 7f685a7fc700 1 --2- 172.21.15.149:0/4212051694 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f684402ff10 0x7f68440323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f684c0096a0 tx=0x7f684c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:31.283 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.281+0000 7f68437fe700 1 -- 172.21.15.149:0/4212051694 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6850014420 con 0x7f685c0f5410 2024-02-12T16:40:31.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:31 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:29.524558+0000 mgr.smithi104.ohwbhf (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:31.484 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.482+0000 7f68613c6700 1 -- 172.21.15.149:0/4212051694 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f685c0fafa0 con 0x7f685c0f5410 2024-02-12T16:40:31.485 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.483+0000 7f68437fe700 1 -- 172.21.15.149:0/4212051694 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6850010230 con 0x7f685c0f5410 2024-02-12T16:40:31.485 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:31.485 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:31.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.484+0000 7f68613c6700 1 -- 172.21.15.149:0/4212051694 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f684402ff10 msgr2=0x7f68440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:31.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.484+0000 7f68613c6700 1 --2- 172.21.15.149:0/4212051694 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f684402ff10 0x7f68440323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f684c0096a0 tx=0x7f684c006b40).stop 2024-02-12T16:40:31.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.484+0000 7f68613c6700 1 -- 172.21.15.149:0/4212051694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685c0f5410 msgr2=0x7f685c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:31.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.484+0000 7f68613c6700 1 --2- 172.21.15.149:0/4212051694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685c0f5410 0x7f685c106810 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f6850005010 tx=0x7f685000b040).stop 2024-02-12T16:40:31.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.484+0000 7f68613c6700 1 -- 172.21.15.149:0/4212051694 shutdown_connections 2024-02-12T16:40:31.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.484+0000 7f68613c6700 1 --2- 172.21.15.149:0/4212051694 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f684402ff10 0x7f68440323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:31.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.484+0000 7f68613c6700 1 --2- 172.21.15.149:0/4212051694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f685c0f5410 0x7f685c106810 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:31.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.484+0000 7f68613c6700 1 -- 172.21.15.149:0/4212051694 >> 172.21.15.149:0/4212051694 conn(0x7f685c0f0b70 msgr2=0x7f685c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:31.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.485+0000 7f68613c6700 1 -- 172.21.15.149:0/4212051694 shutdown_connections 2024-02-12T16:40:31.487 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:31.485+0000 7f68613c6700 1 -- 172.21.15.149:0/4212051694 wait complete. 2024-02-12T16:40:31.488 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:32.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:32 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:31.484378+0000 mon.smithi104 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.149:0/4212051694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:33.217 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:33.217 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:33.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:33 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:31.524804+0000 mgr.smithi104.ohwbhf (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:35.062 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.060+0000 7f3ae50ee700 1 -- 172.21.15.149:0/1486168352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ae00f5410 msgr2=0x7f3ae00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:35.063 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.060+0000 7f3ae50ee700 1 --2- 172.21.15.149:0/1486168352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ae00f5410 0x7f3ae00f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f3ac8009fb0 tx=0x7f3ac8009b30).stop 2024-02-12T16:40:35.063 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.060+0000 7f3ae50ee700 1 -- 172.21.15.149:0/1486168352 shutdown_connections 2024-02-12T16:40:35.063 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.060+0000 7f3ae50ee700 1 --2- 172.21.15.149:0/1486168352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ae00f5410 0x7f3ae00f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:35.063 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.060+0000 7f3ae50ee700 1 -- 172.21.15.149:0/1486168352 >> 172.21.15.149:0/1486168352 conn(0x7f3ae00f0b70 msgr2=0x7f3ae00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:35.063 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.060+0000 7f3ae50ee700 1 -- 172.21.15.149:0/1486168352 shutdown_connections 2024-02-12T16:40:35.063 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.060+0000 7f3ae50ee700 1 -- 172.21.15.149:0/1486168352 wait complete. 2024-02-12T16:40:35.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.061+0000 7f3ae50ee700 1 Processor -- start 2024-02-12T16:40:35.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.061+0000 7f3ae50ee700 1 -- start start 2024-02-12T16:40:35.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.061+0000 7f3ae50ee700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ae00f5410 0x7f3ae01089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:35.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.061+0000 7f3ae50ee700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae0108ee0 con 0x7f3ae00f5410 2024-02-12T16:40:35.065 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.061+0000 7f3aded9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ae00f5410 0x7f3ae01089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.061+0000 7f3aded9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ae00f5410 0x7f3ae01089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:52712/0 (socket says 172.21.15.149:52712) 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.061+0000 7f3aded9d700 1 -- 172.21.15.149:0/2538767177 learned_addr learned my addr 172.21.15.149:0/2538767177 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.062+0000 7f3aded9d700 1 -- 172.21.15.149:0/2538767177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac8005040 con 0x7f3ae00f5410 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.062+0000 7f3aded9d700 1 --2- 172.21.15.149:0/2538767177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ae00f5410 0x7f3ae01089f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3ac8005010 tx=0x7f3ac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.062+0000 7f3ad77fe700 1 -- 172.21.15.149:0/2538767177 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ac8009320 con 0x7f3ae00f5410 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.062+0000 7f3ad77fe700 1 -- 172.21.15.149:0/2538767177 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ac8013380 con 0x7f3ae00f5410 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.062+0000 7f3ae50ee700 1 -- 172.21.15.149:0/2538767177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae01090e0 con 0x7f3ae00f5410 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.062+0000 7f3ad77fe700 1 -- 172.21.15.149:0/2538767177 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ac8014e50 con 0x7f3ae00f5410 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.062+0000 7f3ae50ee700 1 -- 172.21.15.149:0/2538767177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae0109570 con 0x7f3ae00f5410 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.063+0000 7f3ad77fe700 1 -- 172.21.15.149:0/2538767177 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ac8014420 con 0x7f3ae00f5410 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.063+0000 7f3ae50ee700 1 -- 172.21.15.149:0/2538767177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae0045ad0 con 0x7f3ae00f5410 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.063+0000 7f3ad77fe700 1 --2- 172.21.15.149:0/2538767177 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3acc02ff10 0x7f3acc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:35.066 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.063+0000 7f3ad77fe700 1 -- 172.21.15.149:0/2538767177 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ac8040e30 con 0x7f3ae00f5410 2024-02-12T16:40:35.067 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.063+0000 7f3ade59c700 1 --2- 172.21.15.149:0/2538767177 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3acc02ff10 0x7f3acc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:35.067 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.063+0000 7f3ade59c700 1 --2- 172.21.15.149:0/2538767177 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3acc02ff10 0x7f3acc0323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3ad00096a0 tx=0x7f3ad0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:35.069 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.067+0000 7f3ad77fe700 1 -- 172.21.15.149:0/2538767177 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ac80095f0 con 0x7f3ae00f5410 2024-02-12T16:40:35.273 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.271+0000 7f3ae50ee700 1 -- 172.21.15.149:0/2538767177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ae00fafa0 con 0x7f3ae00f5410 2024-02-12T16:40:35.273 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.271+0000 7f3ad77fe700 1 -- 172.21.15.149:0/2538767177 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ac801db70 con 0x7f3ae00f5410 2024-02-12T16:40:35.273 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:35.274 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:35.275 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.273+0000 7f3ae50ee700 1 -- 172.21.15.149:0/2538767177 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3acc02ff10 msgr2=0x7f3acc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:35.275 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.273+0000 7f3ae50ee700 1 --2- 172.21.15.149:0/2538767177 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3acc02ff10 0x7f3acc0323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3ad00096a0 tx=0x7f3ad0006b40).stop 2024-02-12T16:40:35.275 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.273+0000 7f3ae50ee700 1 -- 172.21.15.149:0/2538767177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ae00f5410 msgr2=0x7f3ae01089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:35.275 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.273+0000 7f3ae50ee700 1 --2- 172.21.15.149:0/2538767177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ae00f5410 0x7f3ae01089f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3ac8005010 tx=0x7f3ac800b040).stop 2024-02-12T16:40:35.275 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.273+0000 7f3ae50ee700 1 -- 172.21.15.149:0/2538767177 shutdown_connections 2024-02-12T16:40:35.275 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.273+0000 7f3ae50ee700 1 --2- 172.21.15.149:0/2538767177 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f3acc02ff10 0x7f3acc0323d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:35.275 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.273+0000 7f3ae50ee700 1 --2- 172.21.15.149:0/2538767177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ae00f5410 0x7f3ae01089f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:35.275 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.273+0000 7f3ae50ee700 1 -- 172.21.15.149:0/2538767177 >> 172.21.15.149:0/2538767177 conn(0x7f3ae00f0b70 msgr2=0x7f3ae005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:35.276 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.273+0000 7f3ae50ee700 1 -- 172.21.15.149:0/2538767177 shutdown_connections 2024-02-12T16:40:35.276 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:35.273+0000 7f3ae50ee700 1 -- 172.21.15.149:0/2538767177 wait complete. 2024-02-12T16:40:35.276 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:35.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:35 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:33.525089+0000 mgr.smithi104.ohwbhf (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:36.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:36 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:35.272485+0000 mon.smithi104 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.149:0/2538767177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:36.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:36.884 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:37.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:37 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:35.525452+0000 mgr.smithi104.ohwbhf (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:38.752 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.748+0000 7feda5cb2700 1 -- 172.21.15.149:0/4164498728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda00f5410 msgr2=0x7feda00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:38.752 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.748+0000 7feda5cb2700 1 --2- 172.21.15.149:0/4164498728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda00f5410 0x7feda00f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fed90009fb0 tx=0x7fed90009b30).stop 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.748+0000 7feda5cb2700 1 -- 172.21.15.149:0/4164498728 shutdown_connections 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.748+0000 7feda5cb2700 1 --2- 172.21.15.149:0/4164498728 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda00f5410 0x7feda00f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.748+0000 7feda5cb2700 1 -- 172.21.15.149:0/4164498728 >> 172.21.15.149:0/4164498728 conn(0x7feda00f0b70 msgr2=0x7feda00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.749+0000 7feda5cb2700 1 -- 172.21.15.149:0/4164498728 shutdown_connections 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.749+0000 7feda5cb2700 1 -- 172.21.15.149:0/4164498728 wait complete. 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.749+0000 7feda5cb2700 1 Processor -- start 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.749+0000 7feda5cb2700 1 -- start start 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.749+0000 7feda5cb2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda00f5410 0x7feda0109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.749+0000 7feda5cb2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda0060220 con 0x7feda00f5410 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.750+0000 7fed9f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda00f5410 0x7feda0109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.750+0000 7fed9f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda00f5410 0x7feda0109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:52740/0 (socket says 172.21.15.149:52740) 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.750+0000 7fed9f7fe700 1 -- 172.21.15.149:0/2426325558 learned_addr learned my addr 172.21.15.149:0/2426325558 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.750+0000 7fed9f7fe700 1 -- 172.21.15.149:0/2426325558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed90005040 con 0x7feda00f5410 2024-02-12T16:40:38.753 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.750+0000 7fed9f7fe700 1 --2- 172.21.15.149:0/2426325558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda00f5410 0x7feda0109500 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fed900089f0 tx=0x7fed9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:38.754 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.750+0000 7fed87fff700 1 -- 172.21.15.149:0/2426325558 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed90009320 con 0x7feda00f5410 2024-02-12T16:40:38.754 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.750+0000 7feda5cb2700 1 -- 172.21.15.149:0/2426325558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda0060420 con 0x7feda00f5410 2024-02-12T16:40:38.754 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.750+0000 7fed87fff700 1 -- 172.21.15.149:0/2426325558 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed90013380 con 0x7feda00f5410 2024-02-12T16:40:38.754 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.750+0000 7fed87fff700 1 -- 172.21.15.149:0/2426325558 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed9001ed80 con 0x7feda00f5410 2024-02-12T16:40:38.754 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.750+0000 7feda5cb2700 1 -- 172.21.15.149:0/2426325558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda0109bb0 con 0x7feda00f5410 2024-02-12T16:40:38.755 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.751+0000 7fed87fff700 1 -- 172.21.15.149:0/2426325558 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed9001e420 con 0x7feda00f5410 2024-02-12T16:40:38.755 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.751+0000 7feda5cb2700 1 -- 172.21.15.149:0/2426325558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda0059100 con 0x7feda00f5410 2024-02-12T16:40:38.755 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.751+0000 7fed87fff700 1 --2- 172.21.15.149:0/2426325558 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed8802ff60 0x7fed88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:38.755 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.751+0000 7fed87fff700 1 -- 172.21.15.149:0/2426325558 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed90040ef0 con 0x7feda00f5410 2024-02-12T16:40:38.755 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.751+0000 7fed9effd700 1 --2- 172.21.15.149:0/2426325558 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed8802ff60 0x7fed88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:38.755 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.752+0000 7fed9effd700 1 --2- 172.21.15.149:0/2426325558 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed8802ff60 0x7fed88032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fed940096a0 tx=0x7fed94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:38.758 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.756+0000 7fed87fff700 1 -- 172.21.15.149:0/2426325558 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed9000e2a0 con 0x7feda00f5410 2024-02-12T16:40:38.962 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.960+0000 7feda5cb2700 1 -- 172.21.15.149:0/2426325558 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feda0045ad0 con 0x7feda00f5410 2024-02-12T16:40:38.963 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.960+0000 7fed87fff700 1 -- 172.21.15.149:0/2426325558 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fed9000e2a0 con 0x7feda00f5410 2024-02-12T16:40:38.963 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:38.963 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:38.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.962+0000 7feda5cb2700 1 -- 172.21.15.149:0/2426325558 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed8802ff60 msgr2=0x7fed88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:38.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.962+0000 7feda5cb2700 1 --2- 172.21.15.149:0/2426325558 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed8802ff60 0x7fed88032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fed940096a0 tx=0x7fed94006b40).stop 2024-02-12T16:40:38.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.962+0000 7feda5cb2700 1 -- 172.21.15.149:0/2426325558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda00f5410 msgr2=0x7feda0109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:38.964 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.962+0000 7feda5cb2700 1 --2- 172.21.15.149:0/2426325558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda00f5410 0x7feda0109500 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fed900089f0 tx=0x7fed9000b040).stop 2024-02-12T16:40:38.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.962+0000 7feda5cb2700 1 -- 172.21.15.149:0/2426325558 shutdown_connections 2024-02-12T16:40:38.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.962+0000 7feda5cb2700 1 --2- 172.21.15.149:0/2426325558 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fed8802ff60 0x7fed88032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:38.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.962+0000 7feda5cb2700 1 --2- 172.21.15.149:0/2426325558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda00f5410 0x7feda0109500 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:38.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.962+0000 7feda5cb2700 1 -- 172.21.15.149:0/2426325558 >> 172.21.15.149:0/2426325558 conn(0x7feda00f0b70 msgr2=0x7feda00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:38.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.962+0000 7feda5cb2700 1 -- 172.21.15.149:0/2426325558 shutdown_connections 2024-02-12T16:40:38.965 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:38.962+0000 7feda5cb2700 1 -- 172.21.15.149:0/2426325558 wait complete. 2024-02-12T16:40:38.965 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:39.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:39 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:37.525757+0000 mgr.smithi104.ohwbhf (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:39.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:39 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:38.961648+0000 mon.smithi104 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.149:0/2426325558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:40.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:40.585 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:41.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:41 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:39.526141+0000 mgr.smithi104.ohwbhf (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:42.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.401+0000 7fae5ffff700 1 -- 172.21.15.149:0/1604090340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae600f46c0 msgr2=0x7fae600f4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:42.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.401+0000 7fae5ffff700 1 --2- 172.21.15.149:0/1604090340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae600f46c0 0x7fae600f4b00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fae50009fb0 tx=0x7fae50009b30).stop 2024-02-12T16:40:42.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.401+0000 7fae5ffff700 1 -- 172.21.15.149:0/1604090340 shutdown_connections 2024-02-12T16:40:42.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.401+0000 7fae5ffff700 1 --2- 172.21.15.149:0/1604090340 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae600f46c0 0x7fae600f4b00 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:42.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.401+0000 7fae5ffff700 1 -- 172.21.15.149:0/1604090340 >> 172.21.15.149:0/1604090340 conn(0x7fae600f0120 msgr2=0x7fae600f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:42.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.401+0000 7fae5ffff700 1 -- 172.21.15.149:0/1604090340 shutdown_connections 2024-02-12T16:40:42.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.401+0000 7fae5ffff700 1 -- 172.21.15.149:0/1604090340 wait complete. 2024-02-12T16:40:42.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.401+0000 7fae5ffff700 1 Processor -- start 2024-02-12T16:40:42.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.402+0000 7fae5ffff700 1 -- start start 2024-02-12T16:40:42.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.402+0000 7fae5ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae600f46c0 0x7fae600f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:42.405 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.402+0000 7fae5ffff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae600f5730 con 0x7fae600f46c0 2024-02-12T16:40:42.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.402+0000 7fae5effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae600f46c0 0x7fae600f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:42.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.402+0000 7fae5effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae600f46c0 0x7fae600f72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:52758/0 (socket says 172.21.15.149:52758) 2024-02-12T16:40:42.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.402+0000 7fae5effd700 1 -- 172.21.15.149:0/3070976950 learned_addr learned my addr 172.21.15.149:0/3070976950 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:42.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.402+0000 7fae5effd700 1 -- 172.21.15.149:0/3070976950 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae50005040 con 0x7fae600f46c0 2024-02-12T16:40:42.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.402+0000 7fae5effd700 1 --2- 172.21.15.149:0/3070976950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae600f46c0 0x7fae600f72b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fae50005010 tx=0x7fae5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:42.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.403+0000 7fae477fe700 1 -- 172.21.15.149:0/3070976950 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae50009320 con 0x7fae600f46c0 2024-02-12T16:40:42.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.403+0000 7fae5ffff700 1 -- 172.21.15.149:0/3070976950 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae600f58d0 con 0x7fae600f46c0 2024-02-12T16:40:42.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.403+0000 7fae477fe700 1 -- 172.21.15.149:0/3070976950 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae50013380 con 0x7fae600f46c0 2024-02-12T16:40:42.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.403+0000 7fae477fe700 1 -- 172.21.15.149:0/3070976950 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae50014e50 con 0x7fae600f46c0 2024-02-12T16:40:42.406 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.403+0000 7fae5ffff700 1 -- 172.21.15.149:0/3070976950 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae600f5d60 con 0x7fae600f46c0 2024-02-12T16:40:42.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.403+0000 7fae477fe700 1 -- 172.21.15.149:0/3070976950 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae50014420 con 0x7fae600f46c0 2024-02-12T16:40:42.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.403+0000 7fae5ffff700 1 -- 172.21.15.149:0/3070976950 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae60059100 con 0x7fae600f46c0 2024-02-12T16:40:42.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.404+0000 7fae477fe700 1 --2- 172.21.15.149:0/3070976950 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fae4802ff10 0x7fae480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:42.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.404+0000 7fae477fe700 1 -- 172.21.15.149:0/3070976950 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae50040f40 con 0x7fae600f46c0 2024-02-12T16:40:42.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.404+0000 7fae5e7fc700 1 --2- 172.21.15.149:0/3070976950 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fae4802ff10 0x7fae480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:42.407 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.404+0000 7fae5e7fc700 1 --2- 172.21.15.149:0/3070976950 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fae4802ff10 0x7fae480323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fae540096a0 tx=0x7fae54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:42.410 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.408+0000 7fae477fe700 1 -- 172.21.15.149:0/3070976950 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae50012330 con 0x7fae600f46c0 2024-02-12T16:40:42.613 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.611+0000 7fae5ffff700 1 -- 172.21.15.149:0/3070976950 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae600fbe20 con 0x7fae600f46c0 2024-02-12T16:40:42.613 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.611+0000 7fae477fe700 1 -- 172.21.15.149:0/3070976950 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fae5001db70 con 0x7fae600f46c0 2024-02-12T16:40:42.614 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:42.614 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:42.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.613+0000 7fae5ffff700 1 -- 172.21.15.149:0/3070976950 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fae4802ff10 msgr2=0x7fae480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:42.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.613+0000 7fae5ffff700 1 --2- 172.21.15.149:0/3070976950 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fae4802ff10 0x7fae480323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fae540096a0 tx=0x7fae54006b40).stop 2024-02-12T16:40:42.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.613+0000 7fae5ffff700 1 -- 172.21.15.149:0/3070976950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae600f46c0 msgr2=0x7fae600f72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:42.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.613+0000 7fae5ffff700 1 --2- 172.21.15.149:0/3070976950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae600f46c0 0x7fae600f72b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fae50005010 tx=0x7fae5000b040).stop 2024-02-12T16:40:42.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.613+0000 7fae5ffff700 1 -- 172.21.15.149:0/3070976950 shutdown_connections 2024-02-12T16:40:42.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.613+0000 7fae5ffff700 1 --2- 172.21.15.149:0/3070976950 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fae4802ff10 0x7fae480323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:42.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.613+0000 7fae5ffff700 1 --2- 172.21.15.149:0/3070976950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae600f46c0 0x7fae600f72b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:42.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.613+0000 7fae5ffff700 1 -- 172.21.15.149:0/3070976950 >> 172.21.15.149:0/3070976950 conn(0x7fae600f0120 msgr2=0x7fae600f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:42.615 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.613+0000 7fae5ffff700 1 -- 172.21.15.149:0/3070976950 shutdown_connections 2024-02-12T16:40:42.616 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:42.613+0000 7fae5ffff700 1 -- 172.21.15.149:0/3070976950 wait complete. 2024-02-12T16:40:42.616 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:43.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:43 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:41.526423+0000 mgr.smithi104.ohwbhf (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:43.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:43 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:42.612660+0000 mon.smithi104 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.149:0/3070976950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:44.201 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:44.202 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:45.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:45 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:43.526711+0000 mgr.smithi104.ohwbhf (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:46.199 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.195+0000 7f5d2d692700 1 -- 172.21.15.149:0/1199294342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280f62a0 msgr2=0x7f5d280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:46.199 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.195+0000 7f5d2d692700 1 --2- 172.21.15.149:0/1199294342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280f62a0 0x7f5d280f66c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f5d18009fb0 tx=0x7f5d18009b30).stop 2024-02-12T16:40:46.199 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.196+0000 7f5d2d692700 1 -- 172.21.15.149:0/1199294342 shutdown_connections 2024-02-12T16:40:46.199 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.196+0000 7f5d2d692700 1 --2- 172.21.15.149:0/1199294342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280f62a0 0x7f5d280f66c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:46.199 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.196+0000 7f5d2d692700 1 -- 172.21.15.149:0/1199294342 >> 172.21.15.149:0/1199294342 conn(0x7f5d280f1aa0 msgr2=0x7f5d280f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.196+0000 7f5d2d692700 1 -- 172.21.15.149:0/1199294342 shutdown_connections 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.196+0000 7f5d2d692700 1 -- 172.21.15.149:0/1199294342 wait complete. 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.196+0000 7f5d2d692700 1 Processor -- start 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.196+0000 7f5d2d692700 1 -- start start 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.196+0000 7f5d2d692700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280f62a0 0x7f5d281087e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.196+0000 7f5d2d692700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d28108cd0 con 0x7f5d280f62a0 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.197+0000 7f5d27fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280f62a0 0x7f5d281087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.197+0000 7f5d27fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280f62a0 0x7f5d281087e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:48132/0 (socket says 172.21.15.149:48132) 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.197+0000 7f5d27fff700 1 -- 172.21.15.149:0/391657479 learned_addr learned my addr 172.21.15.149:0/391657479 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.197+0000 7f5d27fff700 1 -- 172.21.15.149:0/391657479 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d18005040 con 0x7f5d280f62a0 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.197+0000 7f5d27fff700 1 --2- 172.21.15.149:0/391657479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280f62a0 0x7f5d281087e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f5d180089f0 tx=0x7f5d1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.197+0000 7f5d24ff9700 1 -- 172.21.15.149:0/391657479 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d18009320 con 0x7f5d280f62a0 2024-02-12T16:40:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.197+0000 7f5d2d692700 1 -- 172.21.15.149:0/391657479 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d28108ed0 con 0x7f5d280f62a0 2024-02-12T16:40:46.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.197+0000 7f5d24ff9700 1 -- 172.21.15.149:0/391657479 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d18013380 con 0x7f5d280f62a0 2024-02-12T16:40:46.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.197+0000 7f5d24ff9700 1 -- 172.21.15.149:0/391657479 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d18014e20 con 0x7f5d280f62a0 2024-02-12T16:40:46.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.197+0000 7f5d2d692700 1 -- 172.21.15.149:0/391657479 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d28109360 con 0x7f5d280f62a0 2024-02-12T16:40:46.201 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.198+0000 7f5d24ff9700 1 -- 172.21.15.149:0/391657479 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d180095f0 con 0x7f5d280f62a0 2024-02-12T16:40:46.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.198+0000 7f5d2d692700 1 -- 172.21.15.149:0/391657479 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d28045ad0 con 0x7f5d280f62a0 2024-02-12T16:40:46.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.198+0000 7f5d24ff9700 1 --2- 172.21.15.149:0/391657479 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff10 0x7f5d100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:46.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.198+0000 7f5d24ff9700 1 -- 172.21.15.149:0/391657479 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d18040f50 con 0x7f5d280f62a0 2024-02-12T16:40:46.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.198+0000 7f5d277fe700 1 --2- 172.21.15.149:0/391657479 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff10 0x7f5d100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:46.202 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.199+0000 7f5d277fe700 1 --2- 172.21.15.149:0/391657479 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff10 0x7f5d100323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f5d1c0096a0 tx=0x7f5d1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:46.205 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.203+0000 7f5d24ff9700 1 -- 172.21.15.149:0/391657479 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d18014420 con 0x7f5d280f62a0 2024-02-12T16:40:46.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.411+0000 7f5d2d692700 1 -- 172.21.15.149:0/391657479 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d2810a210 con 0x7f5d280f62a0 2024-02-12T16:40:46.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.411+0000 7f5d24ff9700 1 -- 172.21.15.149:0/391657479 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5d1801d980 con 0x7f5d280f62a0 2024-02-12T16:40:46.414 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:46.414 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:46.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.413+0000 7f5d2d692700 1 -- 172.21.15.149:0/391657479 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff10 msgr2=0x7f5d100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:46.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.413+0000 7f5d2d692700 1 --2- 172.21.15.149:0/391657479 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff10 0x7f5d100323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f5d1c0096a0 tx=0x7f5d1c006b40).stop 2024-02-12T16:40:46.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.413+0000 7f5d2d692700 1 -- 172.21.15.149:0/391657479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280f62a0 msgr2=0x7f5d281087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:46.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.413+0000 7f5d2d692700 1 --2- 172.21.15.149:0/391657479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280f62a0 0x7f5d281087e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f5d180089f0 tx=0x7f5d1800b040).stop 2024-02-12T16:40:46.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.413+0000 7f5d2d692700 1 -- 172.21.15.149:0/391657479 shutdown_connections 2024-02-12T16:40:46.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.413+0000 7f5d2d692700 1 --2- 172.21.15.149:0/391657479 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff10 0x7f5d100323d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:46.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.413+0000 7f5d2d692700 1 --2- 172.21.15.149:0/391657479 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280f62a0 0x7f5d281087e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:46.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.413+0000 7f5d2d692700 1 -- 172.21.15.149:0/391657479 >> 172.21.15.149:0/391657479 conn(0x7f5d280f1aa0 msgr2=0x7f5d280f2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:46.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.414+0000 7f5d2d692700 1 -- 172.21.15.149:0/391657479 shutdown_connections 2024-02-12T16:40:46.416 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:46.414+0000 7f5d2d692700 1 -- 172.21.15.149:0/391657479 wait complete. 2024-02-12T16:40:46.417 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:47.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:47 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:45.527075+0000 mgr.smithi104.ohwbhf (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:47.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:47 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:46.412794+0000 mon.smithi104 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.149:0/391657479' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:48.111 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:48.112 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:49.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:49 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:47.527404+0000 mgr.smithi104.ohwbhf (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:49.971 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.949+0000 7f74b0305700 1 -- 172.21.15.149:0/3721575498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74a80f5410 msgr2=0x7f74a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:49.972 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.949+0000 7f74b0305700 1 --2- 172.21.15.149:0/3721575498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74a80f5410 0x7f74a80f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f7498009fb0 tx=0x7f7498009b30).stop 2024-02-12T16:40:49.972 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.949+0000 7f74b0305700 1 -- 172.21.15.149:0/3721575498 shutdown_connections 2024-02-12T16:40:49.972 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.949+0000 7f74b0305700 1 --2- 172.21.15.149:0/3721575498 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74a80f5410 0x7f74a80f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:49.972 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.949+0000 7f74b0305700 1 -- 172.21.15.149:0/3721575498 >> 172.21.15.149:0/3721575498 conn(0x7f74a80f0b70 msgr2=0x7f74a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:49.972 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.949+0000 7f74b0305700 1 -- 172.21.15.149:0/3721575498 shutdown_connections 2024-02-12T16:40:49.972 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.949+0000 7f74b0305700 1 -- 172.21.15.149:0/3721575498 wait complete. 2024-02-12T16:40:49.972 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.950+0000 7f74b0305700 1 Processor -- start 2024-02-12T16:40:49.972 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.950+0000 7f74b0305700 1 -- start start 2024-02-12T16:40:49.972 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.950+0000 7f74b0305700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74a80f5410 0x7f74a8109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:49.972 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.950+0000 7f74b0305700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a8060220 con 0x7f74a80f5410 2024-02-12T16:40:49.972 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.950+0000 7f74ae0a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74a80f5410 0x7f74a8109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.950+0000 7f74ae0a1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74a80f5410 0x7f74a8109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:48148/0 (socket says 172.21.15.149:48148) 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.950+0000 7f74ae0a1700 1 -- 172.21.15.149:0/1630388809 learned_addr learned my addr 172.21.15.149:0/1630388809 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.951+0000 7f74ae0a1700 1 -- 172.21.15.149:0/1630388809 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7498005040 con 0x7f74a80f5410 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.951+0000 7f74ae0a1700 1 --2- 172.21.15.149:0/1630388809 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74a80f5410 0x7f74a8109500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f74980089f0 tx=0x7f749800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.951+0000 7f749effd700 1 -- 172.21.15.149:0/1630388809 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7498009320 con 0x7f74a80f5410 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.951+0000 7f74b0305700 1 -- 172.21.15.149:0/1630388809 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a8060420 con 0x7f74a80f5410 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.951+0000 7f749effd700 1 -- 172.21.15.149:0/1630388809 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7498013380 con 0x7f74a80f5410 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.951+0000 7f749effd700 1 -- 172.21.15.149:0/1630388809 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f749801ed80 con 0x7f74a80f5410 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.951+0000 7f74b0305700 1 -- 172.21.15.149:0/1630388809 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a8109bb0 con 0x7f74a80f5410 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.952+0000 7f74b0305700 1 -- 172.21.15.149:0/1630388809 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a8059100 con 0x7f74a80f5410 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.952+0000 7f749effd700 1 -- 172.21.15.149:0/1630388809 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f749801e420 con 0x7f74a80f5410 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.952+0000 7f749effd700 1 --2- 172.21.15.149:0/1630388809 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f749402ff60 0x7f7494032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:49.973 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.952+0000 7f749effd700 1 -- 172.21.15.149:0/1630388809 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7498041090 con 0x7f74a80f5410 2024-02-12T16:40:49.974 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.952+0000 7f74ad8a0700 1 --2- 172.21.15.149:0/1630388809 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f749402ff60 0x7f7494032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:49.974 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.953+0000 7f74ad8a0700 1 --2- 172.21.15.149:0/1630388809 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f749402ff60 0x7f7494032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f74a40096a0 tx=0x7f74a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:49.974 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:49.957+0000 7f749effd700 1 -- 172.21.15.149:0/1630388809 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74980095f0 con 0x7f74a80f5410 2024-02-12T16:40:50.163 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.161+0000 7f74b0305700 1 -- 172.21.15.149:0/1630388809 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74a8045ad0 con 0x7f74a80f5410 2024-02-12T16:40:50.164 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.163+0000 7f749effd700 1 -- 172.21.15.149:0/1630388809 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7498004490 con 0x7f74a80f5410 2024-02-12T16:40:50.164 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:50.164 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:50.166 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.164+0000 7f74b0305700 1 -- 172.21.15.149:0/1630388809 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f749402ff60 msgr2=0x7f7494032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:50.166 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.164+0000 7f74b0305700 1 --2- 172.21.15.149:0/1630388809 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f749402ff60 0x7f7494032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f74a40096a0 tx=0x7f74a4006b40).stop 2024-02-12T16:40:50.166 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.164+0000 7f74b0305700 1 -- 172.21.15.149:0/1630388809 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74a80f5410 msgr2=0x7f74a8109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:50.166 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.164+0000 7f74b0305700 1 --2- 172.21.15.149:0/1630388809 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74a80f5410 0x7f74a8109500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f74980089f0 tx=0x7f749800b040).stop 2024-02-12T16:40:50.166 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.164+0000 7f74b0305700 1 -- 172.21.15.149:0/1630388809 shutdown_connections 2024-02-12T16:40:50.166 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.164+0000 7f74b0305700 1 --2- 172.21.15.149:0/1630388809 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f749402ff60 0x7f7494032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:50.166 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.164+0000 7f74b0305700 1 --2- 172.21.15.149:0/1630388809 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f74a80f5410 0x7f74a8109500 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:50.166 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.164+0000 7f74b0305700 1 -- 172.21.15.149:0/1630388809 >> 172.21.15.149:0/1630388809 conn(0x7f74a80f0b70 msgr2=0x7f74a80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:50.166 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.164+0000 7f74b0305700 1 -- 172.21.15.149:0/1630388809 shutdown_connections 2024-02-12T16:40:50.166 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:50.164+0000 7f74b0305700 1 -- 172.21.15.149:0/1630388809 wait complete. 2024-02-12T16:40:50.167 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:50.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:50 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:50.163182+0000 mon.smithi104 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.149:0/1630388809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:51.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:51 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:49.527756+0000 mgr.smithi104.ohwbhf (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:51.768 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:51.769 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:53.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:53 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:51.528125+0000 mgr.smithi104.ohwbhf (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:53.718 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.715+0000 7fafec98e700 1 -- 172.21.15.149:0/1451639473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe80f6510 msgr2=0x7fafe80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:53.718 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.715+0000 7fafec98e700 1 --2- 172.21.15.149:0/1451639473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe80f6510 0x7fafe80f6930 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fafdc009fb0 tx=0x7fafdc009b30).stop 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.715+0000 7fafec98e700 1 -- 172.21.15.149:0/1451639473 shutdown_connections 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.715+0000 7fafec98e700 1 --2- 172.21.15.149:0/1451639473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe80f6510 0x7fafe80f6930 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.715+0000 7fafec98e700 1 -- 172.21.15.149:0/1451639473 >> 172.21.15.149:0/1451639473 conn(0x7fafe80f1c70 msgr2=0x7fafe80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.715+0000 7fafec98e700 1 -- 172.21.15.149:0/1451639473 shutdown_connections 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.715+0000 7fafec98e700 1 -- 172.21.15.149:0/1451639473 wait complete. 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.716+0000 7fafec98e700 1 Processor -- start 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.716+0000 7fafec98e700 1 -- start start 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.716+0000 7fafec98e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe80f6510 0x7fafe81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.716+0000 7fafec98e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe8106cd0 con 0x7fafe80f6510 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.716+0000 7fafe659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe80f6510 0x7fafe81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.716+0000 7fafe659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe80f6510 0x7fafe81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:48180/0 (socket says 172.21.15.149:48180) 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.716+0000 7fafe659c700 1 -- 172.21.15.149:0/945702988 learned_addr learned my addr 172.21.15.149:0/945702988 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:53.719 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.717+0000 7fafe659c700 1 -- 172.21.15.149:0/945702988 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc005040 con 0x7fafe80f6510 2024-02-12T16:40:53.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.717+0000 7fafe659c700 1 --2- 172.21.15.149:0/945702988 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe80f6510 0x7fafe81067e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fafdc005010 tx=0x7fafdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:53.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.717+0000 7fafd6ffd700 1 -- 172.21.15.149:0/945702988 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafdc009320 con 0x7fafe80f6510 2024-02-12T16:40:53.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.717+0000 7fafec98e700 1 -- 172.21.15.149:0/945702988 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe8106ed0 con 0x7fafe80f6510 2024-02-12T16:40:53.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.717+0000 7fafd6ffd700 1 -- 172.21.15.149:0/945702988 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fafdc013380 con 0x7fafe80f6510 2024-02-12T16:40:53.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.717+0000 7fafd6ffd700 1 -- 172.21.15.149:0/945702988 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafdc014e50 con 0x7fafe80f6510 2024-02-12T16:40:53.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.717+0000 7fafec98e700 1 -- 172.21.15.149:0/945702988 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe8107360 con 0x7fafe80f6510 2024-02-12T16:40:53.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.718+0000 7fafd6ffd700 1 -- 172.21.15.149:0/945702988 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafdc014420 con 0x7fafe80f6510 2024-02-12T16:40:53.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.718+0000 7fafec98e700 1 -- 172.21.15.149:0/945702988 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe8045ad0 con 0x7fafe80f6510 2024-02-12T16:40:53.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.718+0000 7fafd6ffd700 1 --2- 172.21.15.149:0/945702988 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafd002ff10 0x7fafd00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:53.721 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.718+0000 7fafd6ffd700 1 -- 172.21.15.149:0/945702988 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafdc040f40 con 0x7fafe80f6510 2024-02-12T16:40:53.722 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.718+0000 7fafe5d9b700 1 --2- 172.21.15.149:0/945702988 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafd002ff10 0x7fafd00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:53.722 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.718+0000 7fafe5d9b700 1 --2- 172.21.15.149:0/945702988 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafd002ff10 0x7fafd00323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fafd80096a0 tx=0x7fafd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:53.724 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.723+0000 7fafd6ffd700 1 -- 172.21.15.149:0/945702988 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafdc0095f0 con 0x7fafe80f6510 2024-02-12T16:40:53.933 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.931+0000 7fafec98e700 1 -- 172.21.15.149:0/945702988 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafe8108210 con 0x7fafe80f6510 2024-02-12T16:40:53.934 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.931+0000 7fafd6ffd700 1 -- 172.21.15.149:0/945702988 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fafdc01db70 con 0x7fafe80f6510 2024-02-12T16:40:53.934 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:53.934 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:53.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.933+0000 7fafec98e700 1 -- 172.21.15.149:0/945702988 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafd002ff10 msgr2=0x7fafd00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:53.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.933+0000 7fafec98e700 1 --2- 172.21.15.149:0/945702988 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafd002ff10 0x7fafd00323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fafd80096a0 tx=0x7fafd8006b40).stop 2024-02-12T16:40:53.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.933+0000 7fafec98e700 1 -- 172.21.15.149:0/945702988 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe80f6510 msgr2=0x7fafe81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:53.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.933+0000 7fafec98e700 1 --2- 172.21.15.149:0/945702988 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe80f6510 0x7fafe81067e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fafdc005010 tx=0x7fafdc00b040).stop 2024-02-12T16:40:53.935 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.933+0000 7fafec98e700 1 -- 172.21.15.149:0/945702988 shutdown_connections 2024-02-12T16:40:53.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.933+0000 7fafec98e700 1 --2- 172.21.15.149:0/945702988 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fafd002ff10 0x7fafd00323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:53.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.933+0000 7fafec98e700 1 --2- 172.21.15.149:0/945702988 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe80f6510 0x7fafe81067e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:53.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.933+0000 7fafec98e700 1 -- 172.21.15.149:0/945702988 >> 172.21.15.149:0/945702988 conn(0x7fafe80f1c70 msgr2=0x7fafe80f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:53.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.933+0000 7fafec98e700 1 -- 172.21.15.149:0/945702988 shutdown_connections 2024-02-12T16:40:53.936 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:53.933+0000 7fafec98e700 1 -- 172.21.15.149:0/945702988 wait complete. 2024-02-12T16:40:53.937 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:54.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:54 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:53.932605+0000 mon.smithi104 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.149:0/945702988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:55.667 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:55.668 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:55.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:55 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:53.528434+0000 mgr.smithi104.ohwbhf (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:57.428 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.419+0000 7f5d2e575700 1 -- 172.21.15.149:0/363309946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280601c0 msgr2=0x7f5d280f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:57.428 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.419+0000 7f5d2e575700 1 --2- 172.21.15.149:0/363309946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280601c0 0x7f5d280f91b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f5d18009fb0 tx=0x7f5d18009b30).stop 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.419+0000 7f5d2e575700 1 -- 172.21.15.149:0/363309946 shutdown_connections 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.419+0000 7f5d2e575700 1 --2- 172.21.15.149:0/363309946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280601c0 0x7f5d280f91b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.419+0000 7f5d2e575700 1 -- 172.21.15.149:0/363309946 >> 172.21.15.149:0/363309946 conn(0x7f5d280f0b70 msgr2=0x7f5d280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.419+0000 7f5d2e575700 1 -- 172.21.15.149:0/363309946 shutdown_connections 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.419+0000 7f5d2e575700 1 -- 172.21.15.149:0/363309946 wait complete. 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.420+0000 7f5d2e575700 1 Processor -- start 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.420+0000 7f5d2e575700 1 -- start start 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.420+0000 7f5d2e575700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280601c0 0x7f5d281067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.420+0000 7f5d2e575700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d28106cb0 con 0x7f5d280601c0 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.421+0000 7f5d27fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280601c0 0x7f5d281067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.421+0000 7f5d27fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280601c0 0x7f5d281067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:36634/0 (socket says 172.21.15.149:36634) 2024-02-12T16:40:57.429 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.421+0000 7f5d27fff700 1 -- 172.21.15.149:0/2679924104 learned_addr learned my addr 172.21.15.149:0/2679924104 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.421+0000 7f5d27fff700 1 -- 172.21.15.149:0/2679924104 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d18005040 con 0x7f5d280601c0 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.421+0000 7f5d27fff700 1 --2- 172.21.15.149:0/2679924104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280601c0 0x7f5d281067c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f5d18009830 tx=0x7f5d1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.421+0000 7f5d24ff9700 1 -- 172.21.15.149:0/2679924104 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d18009320 con 0x7f5d280601c0 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.421+0000 7f5d2e575700 1 -- 172.21.15.149:0/2679924104 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d28106eb0 con 0x7f5d280601c0 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.421+0000 7f5d24ff9700 1 -- 172.21.15.149:0/2679924104 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d18013380 con 0x7f5d280601c0 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.421+0000 7f5d24ff9700 1 -- 172.21.15.149:0/2679924104 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d18014420 con 0x7f5d280601c0 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.421+0000 7f5d2e575700 1 -- 172.21.15.149:0/2679924104 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d28107340 con 0x7f5d280601c0 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.422+0000 7f5d24ff9700 1 -- 172.21.15.149:0/2679924104 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d1801e420 con 0x7f5d280601c0 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.422+0000 7f5d24ff9700 1 --2- 172.21.15.149:0/2679924104 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff60 0x7f5d10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.422+0000 7f5d2e575700 1 -- 172.21.15.149:0/2679924104 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d28045ad0 con 0x7f5d280601c0 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.422+0000 7f5d24ff9700 1 -- 172.21.15.149:0/2679924104 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d18040eb0 con 0x7f5d280601c0 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.422+0000 7f5d277fe700 1 --2- 172.21.15.149:0/2679924104 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff60 0x7f5d10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:40:57.430 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.423+0000 7f5d277fe700 1 --2- 172.21.15.149:0/2679924104 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff60 0x7f5d10032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5d1c0096a0 tx=0x7f5d1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:40:57.431 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.427+0000 7f5d24ff9700 1 -- 172.21.15.149:0/2679924104 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d180672b0 con 0x7f5d280601c0 2024-02-12T16:40:57.635 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.633+0000 7f5d2e575700 1 -- 172.21.15.149:0/2679924104 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d280fc090 con 0x7f5d280601c0 2024-02-12T16:40:57.635 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.633+0000 7f5d24ff9700 1 -- 172.21.15.149:0/2679924104 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5d1802c2d0 con 0x7f5d280601c0 2024-02-12T16:40:57.636 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:40:57.636 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:40:57.637 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.635+0000 7f5d2e575700 1 -- 172.21.15.149:0/2679924104 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff60 msgr2=0x7f5d10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:57.637 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.635+0000 7f5d2e575700 1 --2- 172.21.15.149:0/2679924104 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff60 0x7f5d10032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5d1c0096a0 tx=0x7f5d1c006b40).stop 2024-02-12T16:40:57.637 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.635+0000 7f5d2e575700 1 -- 172.21.15.149:0/2679924104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280601c0 msgr2=0x7f5d281067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:40:57.638 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.635+0000 7f5d2e575700 1 --2- 172.21.15.149:0/2679924104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280601c0 0x7f5d281067c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f5d18009830 tx=0x7f5d1800b040).stop 2024-02-12T16:40:57.638 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.635+0000 7f5d2e575700 1 -- 172.21.15.149:0/2679924104 shutdown_connections 2024-02-12T16:40:57.638 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.635+0000 7f5d2e575700 1 --2- 172.21.15.149:0/2679924104 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f5d1002ff60 0x7f5d10032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:57.638 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.635+0000 7f5d2e575700 1 --2- 172.21.15.149:0/2679924104 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d280601c0 0x7f5d281067c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:40:57.638 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.635+0000 7f5d2e575700 1 -- 172.21.15.149:0/2679924104 >> 172.21.15.149:0/2679924104 conn(0x7f5d280f0b70 msgr2=0x7f5d280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:40:57.638 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.635+0000 7f5d2e575700 1 -- 172.21.15.149:0/2679924104 shutdown_connections 2024-02-12T16:40:57.638 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:40:57.635+0000 7f5d2e575700 1 -- 172.21.15.149:0/2679924104 wait complete. 2024-02-12T16:40:57.638 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:40:57.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:57 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:55.528791+0000 mgr.smithi104.ohwbhf (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:40:58.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:58 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:40:57.634759+0000 mon.smithi104 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.149:0/2679924104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:40:59.328 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:40:59.328 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:40:59.676 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:40:59 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:57.529133+0000 mgr.smithi104.ohwbhf (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:01.227 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.222+0000 7f0e30bc2700 1 -- 172.21.15.149:0/2950059934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e2c0f51a0 msgr2=0x7f0e2c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:01.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.222+0000 7f0e30bc2700 1 --2- 172.21.15.149:0/2950059934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e2c0f51a0 0x7f0e2c0f55c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f0e1c009fb0 tx=0x7f0e1c009b30).stop 2024-02-12T16:41:01.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.222+0000 7f0e30bc2700 1 -- 172.21.15.149:0/2950059934 shutdown_connections 2024-02-12T16:41:01.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.222+0000 7f0e30bc2700 1 --2- 172.21.15.149:0/2950059934 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e2c0f51a0 0x7f0e2c0f55c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:01.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.222+0000 7f0e30bc2700 1 -- 172.21.15.149:0/2950059934 >> 172.21.15.149:0/2950059934 conn(0x7f0e2c0f09a0 msgr2=0x7f0e2c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:01.228 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.222+0000 7f0e30bc2700 1 -- 172.21.15.149:0/2950059934 shutdown_connections 2024-02-12T16:41:01.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.222+0000 7f0e30bc2700 1 -- 172.21.15.149:0/2950059934 wait complete. 2024-02-12T16:41:01.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.223+0000 7f0e30bc2700 1 Processor -- start 2024-02-12T16:41:01.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.223+0000 7f0e30bc2700 1 -- start start 2024-02-12T16:41:01.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.223+0000 7f0e30bc2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e2c0f51a0 0x7f0e2c109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:01.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.223+0000 7f0e30bc2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c060280 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.223+0000 7f0e2b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e2c0f51a0 0x7f0e2c109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:01.229 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.223+0000 7f0e2b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e2c0f51a0 0x7f0e2c109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:36640/0 (socket says 172.21.15.149:36640) 2024-02-12T16:41:01.230 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.223+0000 7f0e2b7fe700 1 -- 172.21.15.149:0/1412856325 learned_addr learned my addr 172.21.15.149:0/1412856325 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:41:01.230 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.224+0000 7f0e2b7fe700 1 -- 172.21.15.149:0/1412856325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c005040 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.230 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.224+0000 7f0e2b7fe700 1 --2- 172.21.15.149:0/1412856325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e2c0f51a0 0x7f0e2c109290 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f0e1c009830 tx=0x7f0e1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:01.230 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.224+0000 7f0e28ff9700 1 -- 172.21.15.149:0/1412856325 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1c009320 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.230 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.224+0000 7f0e30bc2700 1 -- 172.21.15.149:0/1412856325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c060480 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.230 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.224+0000 7f0e28ff9700 1 -- 172.21.15.149:0/1412856325 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e1c013380 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.230 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.224+0000 7f0e28ff9700 1 -- 172.21.15.149:0/1412856325 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1c014420 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.231 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.224+0000 7f0e30bc2700 1 -- 172.21.15.149:0/1412856325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c1099c0 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.231 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.225+0000 7f0e28ff9700 1 -- 172.21.15.149:0/1412856325 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e1c01e420 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.231 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.225+0000 7f0e30bc2700 1 -- 172.21.15.149:0/1412856325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c045ad0 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.231 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.225+0000 7f0e28ff9700 1 --2- 172.21.15.149:0/1412856325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0e1802ff60 0x7f0e18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:01.231 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.225+0000 7f0e28ff9700 1 -- 172.21.15.149:0/1412856325 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e1c040e20 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.231 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.225+0000 7f0e23fff700 1 --2- 172.21.15.149:0/1412856325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0e1802ff60 0x7f0e18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:01.231 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.226+0000 7f0e23fff700 1 --2- 172.21.15.149:0/1412856325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0e1802ff60 0x7f0e18032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f0e140096a0 tx=0x7f0e14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:01.232 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.229+0000 7f0e28ff9700 1 -- 172.21.15.149:0/1412856325 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e1c01c930 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.434 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.433+0000 7f0e30bc2700 1 -- 172.21.15.149:0/1412856325 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e2c10a220 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.435 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.434+0000 7f0e28ff9700 1 -- 172.21.15.149:0/1412856325 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0e1c02c2d0 con 0x7f0e2c0f51a0 2024-02-12T16:41:01.435 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:41:01.436 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:41:01.437 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.436+0000 7f0e30bc2700 1 -- 172.21.15.149:0/1412856325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0e1802ff60 msgr2=0x7f0e18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:01.437 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.436+0000 7f0e30bc2700 1 --2- 172.21.15.149:0/1412856325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0e1802ff60 0x7f0e18032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f0e140096a0 tx=0x7f0e14006b40).stop 2024-02-12T16:41:01.437 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.436+0000 7f0e30bc2700 1 -- 172.21.15.149:0/1412856325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e2c0f51a0 msgr2=0x7f0e2c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:01.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.436+0000 7f0e30bc2700 1 --2- 172.21.15.149:0/1412856325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e2c0f51a0 0x7f0e2c109290 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f0e1c009830 tx=0x7f0e1c00b040).stop 2024-02-12T16:41:01.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.436+0000 7f0e30bc2700 1 -- 172.21.15.149:0/1412856325 shutdown_connections 2024-02-12T16:41:01.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.436+0000 7f0e30bc2700 1 --2- 172.21.15.149:0/1412856325 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f0e1802ff60 0x7f0e18032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:01.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.436+0000 7f0e30bc2700 1 --2- 172.21.15.149:0/1412856325 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e2c0f51a0 0x7f0e2c109290 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:01.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.436+0000 7f0e30bc2700 1 -- 172.21.15.149:0/1412856325 >> 172.21.15.149:0/1412856325 conn(0x7f0e2c0f09a0 msgr2=0x7f0e2c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:01.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.436+0000 7f0e30bc2700 1 -- 172.21.15.149:0/1412856325 shutdown_connections 2024-02-12T16:41:01.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:01.436+0000 7f0e30bc2700 1 -- 172.21.15.149:0/1412856325 wait complete. 2024-02-12T16:41:01.438 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:41:01.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:01 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:40:59.529443+0000 mgr.smithi104.ohwbhf (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:02.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:02 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:41:01.434221+0000 mon.smithi104 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.149:0/1412856325' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:41:03.097 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:41:03.097 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:41:03.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:03 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:41:01.529687+0000 mgr.smithi104.ohwbhf (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:05.007 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.005+0000 7fcb7f692700 1 -- 172.21.15.149:0/2990122904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb780f6510 msgr2=0x7fcb780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:05.007 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.005+0000 7fcb7f692700 1 --2- 172.21.15.149:0/2990122904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb780f6510 0x7fcb780f6930 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fcb68009fb0 tx=0x7fcb68009b30).stop 2024-02-12T16:41:05.007 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.005+0000 7fcb7f692700 1 -- 172.21.15.149:0/2990122904 shutdown_connections 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.005+0000 7fcb7f692700 1 --2- 172.21.15.149:0/2990122904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb780f6510 0x7fcb780f6930 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.005+0000 7fcb7f692700 1 -- 172.21.15.149:0/2990122904 >> 172.21.15.149:0/2990122904 conn(0x7fcb780f1c70 msgr2=0x7fcb780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.005+0000 7fcb7f692700 1 -- 172.21.15.149:0/2990122904 shutdown_connections 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.005+0000 7fcb7f692700 1 -- 172.21.15.149:0/2990122904 wait complete. 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.005+0000 7fcb7f692700 1 Processor -- start 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.006+0000 7fcb7f692700 1 -- start start 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.006+0000 7fcb7f692700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb780f6510 0x7fcb781067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.006+0000 7fcb7f692700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb78106cd0 con 0x7fcb780f6510 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.006+0000 7fcb7d42e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb780f6510 0x7fcb781067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.006+0000 7fcb7d42e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb780f6510 0x7fcb781067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:51906/0 (socket says 172.21.15.149:51906) 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.006+0000 7fcb7d42e700 1 -- 172.21.15.149:0/552433986 learned_addr learned my addr 172.21.15.149:0/552433986 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.006+0000 7fcb7d42e700 1 -- 172.21.15.149:0/552433986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb68005040 con 0x7fcb780f6510 2024-02-12T16:41:05.008 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.006+0000 7fcb7d42e700 1 --2- 172.21.15.149:0/552433986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb780f6510 0x7fcb781067e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fcb68005010 tx=0x7fcb6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:05.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.007+0000 7fcb6dffb700 1 -- 172.21.15.149:0/552433986 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb68009320 con 0x7fcb780f6510 2024-02-12T16:41:05.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.007+0000 7fcb7f692700 1 -- 172.21.15.149:0/552433986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb78106ed0 con 0x7fcb780f6510 2024-02-12T16:41:05.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.007+0000 7fcb6dffb700 1 -- 172.21.15.149:0/552433986 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb68013380 con 0x7fcb780f6510 2024-02-12T16:41:05.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.007+0000 7fcb6dffb700 1 -- 172.21.15.149:0/552433986 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb68014e50 con 0x7fcb780f6510 2024-02-12T16:41:05.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.007+0000 7fcb7f692700 1 -- 172.21.15.149:0/552433986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb78107360 con 0x7fcb780f6510 2024-02-12T16:41:05.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.008+0000 7fcb6dffb700 1 -- 172.21.15.149:0/552433986 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb680095f0 con 0x7fcb780f6510 2024-02-12T16:41:05.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.008+0000 7fcb7f692700 1 -- 172.21.15.149:0/552433986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb78045ad0 con 0x7fcb780f6510 2024-02-12T16:41:05.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.008+0000 7fcb6dffb700 1 --2- 172.21.15.149:0/552433986 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcb6402ff10 0x7fcb640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:05.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.008+0000 7fcb6dffb700 1 -- 172.21.15.149:0/552433986 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb68040f80 con 0x7fcb780f6510 2024-02-12T16:41:05.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.008+0000 7fcb7cc2d700 1 --2- 172.21.15.149:0/552433986 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcb6402ff10 0x7fcb640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:05.010 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.009+0000 7fcb7cc2d700 1 --2- 172.21.15.149:0/552433986 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcb6402ff10 0x7fcb640323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fcb740096a0 tx=0x7fcb74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:05.013 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.013+0000 7fcb6dffb700 1 -- 172.21.15.149:0/552433986 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb68014420 con 0x7fcb780f6510 2024-02-12T16:41:05.217 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.216+0000 7fcb7f692700 1 -- 172.21.15.149:0/552433986 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb78108210 con 0x7fcb780f6510 2024-02-12T16:41:05.219 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.218+0000 7fcb6dffb700 1 -- 172.21.15.149:0/552433986 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcb68010230 con 0x7fcb780f6510 2024-02-12T16:41:05.220 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:41:05.220 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:41:05.221 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.220+0000 7fcb7f692700 1 -- 172.21.15.149:0/552433986 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcb6402ff10 msgr2=0x7fcb640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:05.221 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.220+0000 7fcb7f692700 1 --2- 172.21.15.149:0/552433986 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcb6402ff10 0x7fcb640323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fcb740096a0 tx=0x7fcb74006b40).stop 2024-02-12T16:41:05.221 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.220+0000 7fcb7f692700 1 -- 172.21.15.149:0/552433986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb780f6510 msgr2=0x7fcb781067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:05.221 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.220+0000 7fcb7f692700 1 --2- 172.21.15.149:0/552433986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb780f6510 0x7fcb781067e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fcb68005010 tx=0x7fcb6800b040).stop 2024-02-12T16:41:05.222 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.220+0000 7fcb7f692700 1 -- 172.21.15.149:0/552433986 shutdown_connections 2024-02-12T16:41:05.222 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.220+0000 7fcb7f692700 1 --2- 172.21.15.149:0/552433986 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcb6402ff10 0x7fcb640323d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:05.222 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.220+0000 7fcb7f692700 1 --2- 172.21.15.149:0/552433986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb780f6510 0x7fcb781067e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:05.222 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.220+0000 7fcb7f692700 1 -- 172.21.15.149:0/552433986 >> 172.21.15.149:0/552433986 conn(0x7fcb780f1c70 msgr2=0x7fcb780f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:05.222 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.220+0000 7fcb7f692700 1 -- 172.21.15.149:0/552433986 shutdown_connections 2024-02-12T16:41:05.222 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:05.220+0000 7fcb7f692700 1 -- 172.21.15.149:0/552433986 wait complete. 2024-02-12T16:41:05.222 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:41:05.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:05 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:41:03.529969+0000 mgr.smithi104.ohwbhf (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:06.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:06 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:41:05.218768+0000 mon.smithi104 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.149:0/552433986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:41:06.912 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:41:06.912 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:41:07.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:07 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:41:05.530254+0000 mgr.smithi104.ohwbhf (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:08.789 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.787+0000 7fcd9da24700 1 -- 172.21.15.149:0/1714219482 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd980f51a0 msgr2=0x7fcd980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:08.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.787+0000 7fcd9da24700 1 --2- 172.21.15.149:0/1714219482 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd980f51a0 0x7fcd980f55c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fcd8c009fb0 tx=0x7fcd8c009b30).stop 2024-02-12T16:41:08.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.787+0000 7fcd9da24700 1 -- 172.21.15.149:0/1714219482 shutdown_connections 2024-02-12T16:41:08.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.787+0000 7fcd9da24700 1 --2- 172.21.15.149:0/1714219482 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd980f51a0 0x7fcd980f55c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:08.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.787+0000 7fcd9da24700 1 -- 172.21.15.149:0/1714219482 >> 172.21.15.149:0/1714219482 conn(0x7fcd980f09a0 msgr2=0x7fcd980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:08.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.787+0000 7fcd9da24700 1 -- 172.21.15.149:0/1714219482 shutdown_connections 2024-02-12T16:41:08.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.787+0000 7fcd9da24700 1 -- 172.21.15.149:0/1714219482 wait complete. 2024-02-12T16:41:08.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.787+0000 7fcd9da24700 1 Processor -- start 2024-02-12T16:41:08.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.787+0000 7fcd9da24700 1 -- start start 2024-02-12T16:41:08.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.788+0000 7fcd9da24700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd980f51a0 0x7fcd9810b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:08.790 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.788+0000 7fcd9da24700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd981070e0 con 0x7fcd980f51a0 2024-02-12T16:41:08.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.788+0000 7fcd9ca22700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd980f51a0 0x7fcd9810b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:08.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.788+0000 7fcd9ca22700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd980f51a0 0x7fcd9810b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:51922/0 (socket says 172.21.15.149:51922) 2024-02-12T16:41:08.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.788+0000 7fcd9ca22700 1 -- 172.21.15.149:0/3217794477 learned_addr learned my addr 172.21.15.149:0/3217794477 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:41:08.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.788+0000 7fcd9ca22700 1 -- 172.21.15.149:0/3217794477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c005040 con 0x7fcd980f51a0 2024-02-12T16:41:08.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.788+0000 7fcd9ca22700 1 --2- 172.21.15.149:0/3217794477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd980f51a0 0x7fcd9810b6f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fcd8c0089f0 tx=0x7fcd8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:08.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.788+0000 7fcd957fa700 1 -- 172.21.15.149:0/3217794477 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd8c009320 con 0x7fcd980f51a0 2024-02-12T16:41:08.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.788+0000 7fcd9da24700 1 -- 172.21.15.149:0/3217794477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd98060220 con 0x7fcd980f51a0 2024-02-12T16:41:08.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.789+0000 7fcd957fa700 1 -- 172.21.15.149:0/3217794477 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd8c013380 con 0x7fcd980f51a0 2024-02-12T16:41:08.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.789+0000 7fcd957fa700 1 -- 172.21.15.149:0/3217794477 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd8c014e20 con 0x7fcd980f51a0 2024-02-12T16:41:08.792 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.789+0000 7fcd9da24700 1 -- 172.21.15.149:0/3217794477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9810bbe0 con 0x7fcd980f51a0 2024-02-12T16:41:08.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.789+0000 7fcd957fa700 1 -- 172.21.15.149:0/3217794477 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd8c0263a0 con 0x7fcd980f51a0 2024-02-12T16:41:08.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.789+0000 7fcd957fa700 1 --2- 172.21.15.149:0/3217794477 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd8802ff10 0x7fcd880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:08.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.789+0000 7fcd9da24700 1 -- 172.21.15.149:0/3217794477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd98045ad0 con 0x7fcd980f51a0 2024-02-12T16:41:08.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.790+0000 7fcd957fa700 1 -- 172.21.15.149:0/3217794477 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd8c041250 con 0x7fcd980f51a0 2024-02-12T16:41:08.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.790+0000 7fcd97fff700 1 --2- 172.21.15.149:0/3217794477 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd8802ff10 0x7fcd880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:08.793 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.790+0000 7fcd97fff700 1 --2- 172.21.15.149:0/3217794477 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd8802ff10 0x7fcd880323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fcd840096a0 tx=0x7fcd84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:08.796 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.794+0000 7fcd957fa700 1 -- 172.21.15.149:0/3217794477 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd8c01ccf0 con 0x7fcd980f51a0 2024-02-12T16:41:09.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:08.999+0000 7fcd9da24700 1 -- 172.21.15.149:0/3217794477 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd9810c3a0 con 0x7fcd980f51a0 2024-02-12T16:41:09.002 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:09.000+0000 7fcd957fa700 1 -- 172.21.15.149:0/3217794477 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd8c012330 con 0x7fcd980f51a0 2024-02-12T16:41:09.002 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:41:09.002 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:41:09.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:09.001+0000 7fcd9da24700 1 -- 172.21.15.149:0/3217794477 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd8802ff10 msgr2=0x7fcd880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:09.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:09.001+0000 7fcd9da24700 1 --2- 172.21.15.149:0/3217794477 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd8802ff10 0x7fcd880323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fcd840096a0 tx=0x7fcd84006b40).stop 2024-02-12T16:41:09.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:09.001+0000 7fcd9da24700 1 -- 172.21.15.149:0/3217794477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd980f51a0 msgr2=0x7fcd9810b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:09.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:09.001+0000 7fcd9da24700 1 --2- 172.21.15.149:0/3217794477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd980f51a0 0x7fcd9810b6f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fcd8c0089f0 tx=0x7fcd8c00b040).stop 2024-02-12T16:41:09.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:09.002+0000 7fcd9da24700 1 -- 172.21.15.149:0/3217794477 shutdown_connections 2024-02-12T16:41:09.004 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:09.002+0000 7fcd9da24700 1 --2- 172.21.15.149:0/3217794477 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fcd8802ff10 0x7fcd880323d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:09.005 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:09.002+0000 7fcd9da24700 1 --2- 172.21.15.149:0/3217794477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd980f51a0 0x7fcd9810b6f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:09.005 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:09.002+0000 7fcd9da24700 1 -- 172.21.15.149:0/3217794477 >> 172.21.15.149:0/3217794477 conn(0x7fcd980f09a0 msgr2=0x7fcd980f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:09.005 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:09.002+0000 7fcd9da24700 1 -- 172.21.15.149:0/3217794477 shutdown_connections 2024-02-12T16:41:09.005 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:09.002+0000 7fcd9da24700 1 -- 172.21.15.149:0/3217794477 wait complete. 2024-02-12T16:41:09.005 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:41:09.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:41:07.530622+0000 mgr.smithi104.ohwbhf (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:09.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:09 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:41:09.001386+0000 mon.smithi104 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.149:0/3217794477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:41:10.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:41:10.659 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:41:11.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:11 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:41:09.531012+0000 mgr.smithi104.ohwbhf (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:12.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.445+0000 7f6f183dc700 1 -- 172.21.15.149:0/1853330031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f10060280 msgr2=0x7f6f100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:12.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.445+0000 7f6f183dc700 1 --2- 172.21.15.149:0/1853330031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f10060280 0x7f6f100f90f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f6f00009fb0 tx=0x7f6f00009b30).stop 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.445+0000 7f6f183dc700 1 -- 172.21.15.149:0/1853330031 shutdown_connections 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.445+0000 7f6f183dc700 1 --2- 172.21.15.149:0/1853330031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f10060280 0x7f6f100f90f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.445+0000 7f6f183dc700 1 -- 172.21.15.149:0/1853330031 >> 172.21.15.149:0/1853330031 conn(0x7f6f100f0b70 msgr2=0x7f6f100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.445+0000 7f6f183dc700 1 -- 172.21.15.149:0/1853330031 shutdown_connections 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.446+0000 7f6f183dc700 1 -- 172.21.15.149:0/1853330031 wait complete. 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.446+0000 7f6f183dc700 1 Processor -- start 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.446+0000 7f6f183dc700 1 -- start start 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.446+0000 7f6f183dc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f10060280 0x7f6f100f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.446+0000 7f6f183dc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f100f4ff0 con 0x7f6f10060280 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.446+0000 7f6f16178700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f10060280 0x7f6f100f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.446+0000 7f6f16178700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f10060280 0x7f6f100f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:51938/0 (socket says 172.21.15.149:51938) 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.446+0000 7f6f16178700 1 -- 172.21.15.149:0/1495239772 learned_addr learned my addr 172.21.15.149:0/1495239772 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:41:12.450 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.447+0000 7f6f16178700 1 -- 172.21.15.149:0/1495239772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00005040 con 0x7f6f10060280 2024-02-12T16:41:12.451 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.447+0000 7f6f16178700 1 --2- 172.21.15.149:0/1495239772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f10060280 0x7f6f100f6b70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f6f000089f0 tx=0x7f6f0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:12.451 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.447+0000 7f6f06ffd700 1 -- 172.21.15.149:0/1495239772 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f00009320 con 0x7f6f10060280 2024-02-12T16:41:12.451 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.447+0000 7f6f183dc700 1 -- 172.21.15.149:0/1495239772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f100f5190 con 0x7f6f10060280 2024-02-12T16:41:12.451 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.447+0000 7f6f06ffd700 1 -- 172.21.15.149:0/1495239772 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f00013380 con 0x7f6f10060280 2024-02-12T16:41:12.451 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.447+0000 7f6f06ffd700 1 -- 172.21.15.149:0/1495239772 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f00014e20 con 0x7f6f10060280 2024-02-12T16:41:12.451 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.447+0000 7f6f183dc700 1 -- 172.21.15.149:0/1495239772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f100f5620 con 0x7f6f10060280 2024-02-12T16:41:12.452 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.448+0000 7f6f06ffd700 1 -- 172.21.15.149:0/1495239772 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f000095f0 con 0x7f6f10060280 2024-02-12T16:41:12.452 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.448+0000 7f6f183dc700 1 -- 172.21.15.149:0/1495239772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f10059100 con 0x7f6f10060280 2024-02-12T16:41:12.452 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.448+0000 7f6f06ffd700 1 --2- 172.21.15.149:0/1495239772 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6efc02ff10 0x7f6efc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:12.452 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.448+0000 7f6f06ffd700 1 -- 172.21.15.149:0/1495239772 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f00040f10 con 0x7f6f10060280 2024-02-12T16:41:12.452 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.448+0000 7f6f15977700 1 --2- 172.21.15.149:0/1495239772 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6efc02ff10 0x7f6efc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:12.452 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.449+0000 7f6f15977700 1 --2- 172.21.15.149:0/1495239772 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6efc02ff10 0x7f6efc0323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f6f0c0096a0 tx=0x7f6f0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:12.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.453+0000 7f6f06ffd700 1 -- 172.21.15.149:0/1495239772 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f00014420 con 0x7f6f10060280 2024-02-12T16:41:12.659 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.657+0000 7f6f183dc700 1 -- 172.21.15.149:0/1495239772 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f100fc090 con 0x7f6f10060280 2024-02-12T16:41:12.659 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.657+0000 7f6f06ffd700 1 -- 172.21.15.149:0/1495239772 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6f0001d980 con 0x7f6f10060280 2024-02-12T16:41:12.659 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:41:12.660 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:41:12.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.659+0000 7f6f183dc700 1 -- 172.21.15.149:0/1495239772 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6efc02ff10 msgr2=0x7f6efc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:12.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.659+0000 7f6f183dc700 1 --2- 172.21.15.149:0/1495239772 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6efc02ff10 0x7f6efc0323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f6f0c0096a0 tx=0x7f6f0c006b40).stop 2024-02-12T16:41:12.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.659+0000 7f6f183dc700 1 -- 172.21.15.149:0/1495239772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f10060280 msgr2=0x7f6f100f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:12.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.659+0000 7f6f183dc700 1 --2- 172.21.15.149:0/1495239772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f10060280 0x7f6f100f6b70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f6f000089f0 tx=0x7f6f0000b040).stop 2024-02-12T16:41:12.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.659+0000 7f6f183dc700 1 -- 172.21.15.149:0/1495239772 shutdown_connections 2024-02-12T16:41:12.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.659+0000 7f6f183dc700 1 --2- 172.21.15.149:0/1495239772 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f6efc02ff10 0x7f6efc0323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:12.661 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.659+0000 7f6f183dc700 1 --2- 172.21.15.149:0/1495239772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f10060280 0x7f6f100f6b70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:12.662 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.659+0000 7f6f183dc700 1 -- 172.21.15.149:0/1495239772 >> 172.21.15.149:0/1495239772 conn(0x7f6f100f0b70 msgr2=0x7f6f100f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:12.662 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.659+0000 7f6f183dc700 1 -- 172.21.15.149:0/1495239772 shutdown_connections 2024-02-12T16:41:12.662 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:12.659+0000 7f6f183dc700 1 -- 172.21.15.149:0/1495239772 wait complete. 2024-02-12T16:41:12.662 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:41:13.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:41:11.531386+0000 mgr.smithi104.ohwbhf (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:13.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:13 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:41:12.658588+0000 mon.smithi104 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.149:0/1495239772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:41:14.260 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:41:14.261 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:41:15.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:15 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:41:13.531738+0000 mgr.smithi104.ohwbhf (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:16.206 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.202+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2291947609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b8060280 msgr2=0x7f62b80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:16.206 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.202+0000 7f62bf3c9700 1 --2- 172.21.15.149:0/2291947609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b8060280 0x7f62b80f90f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f62a8009fb0 tx=0x7f62a8009b30).stop 2024-02-12T16:41:16.206 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.202+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2291947609 shutdown_connections 2024-02-12T16:41:16.206 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.202+0000 7f62bf3c9700 1 --2- 172.21.15.149:0/2291947609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b8060280 0x7f62b80f90f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:16.206 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.202+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2291947609 >> 172.21.15.149:0/2291947609 conn(0x7f62b80f0b70 msgr2=0x7f62b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:16.206 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.202+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2291947609 shutdown_connections 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.202+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2291947609 wait complete. 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.203+0000 7f62bf3c9700 1 Processor -- start 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.203+0000 7f62bf3c9700 1 -- start start 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.203+0000 7f62bf3c9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b8060280 0x7f62b80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.203+0000 7f62bf3c9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b80f4ff0 con 0x7f62b8060280 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.203+0000 7f62bd165700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b8060280 0x7f62b80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.203+0000 7f62bd165700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b8060280 0x7f62b80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:60812/0 (socket says 172.21.15.149:60812) 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.203+0000 7f62bd165700 1 -- 172.21.15.149:0/2664860766 learned_addr learned my addr 172.21.15.149:0/2664860766 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.203+0000 7f62bd165700 1 -- 172.21.15.149:0/2664860766 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a8005040 con 0x7f62b8060280 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.204+0000 7f62bd165700 1 --2- 172.21.15.149:0/2664860766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b8060280 0x7f62b80f6b70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f62a80089f0 tx=0x7f62a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.204+0000 7f62adffb700 1 -- 172.21.15.149:0/2664860766 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62a8009320 con 0x7f62b8060280 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.204+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2664860766 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b80f5190 con 0x7f62b8060280 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.204+0000 7f62adffb700 1 -- 172.21.15.149:0/2664860766 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62a8013380 con 0x7f62b8060280 2024-02-12T16:41:16.207 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.204+0000 7f62adffb700 1 -- 172.21.15.149:0/2664860766 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62a8014e20 con 0x7f62b8060280 2024-02-12T16:41:16.208 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.204+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2664860766 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b80f5620 con 0x7f62b8060280 2024-02-12T16:41:16.208 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.205+0000 7f62adffb700 1 -- 172.21.15.149:0/2664860766 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62a80095f0 con 0x7f62b8060280 2024-02-12T16:41:16.208 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.205+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2664860766 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b8059100 con 0x7f62b8060280 2024-02-12T16:41:16.208 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.205+0000 7f62adffb700 1 --2- 172.21.15.149:0/2664860766 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f62a402ff10 0x7f62a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:16.209 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.205+0000 7f62adffb700 1 -- 172.21.15.149:0/2664860766 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62a8040f80 con 0x7f62b8060280 2024-02-12T16:41:16.209 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.205+0000 7f62bc964700 1 --2- 172.21.15.149:0/2664860766 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f62a402ff10 0x7f62a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:16.209 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.206+0000 7f62bc964700 1 --2- 172.21.15.149:0/2664860766 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f62a402ff10 0x7f62a40323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f62b40096a0 tx=0x7f62b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:16.212 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.210+0000 7f62adffb700 1 -- 172.21.15.149:0/2664860766 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62a8014420 con 0x7f62b8060280 2024-02-12T16:41:16.420 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.417+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2664860766 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62b8045ad0 con 0x7f62b8060280 2024-02-12T16:41:16.420 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.418+0000 7f62adffb700 1 -- 172.21.15.149:0/2664860766 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f62a8010230 con 0x7f62b8060280 2024-02-12T16:41:16.420 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:41:16.421 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:41:16.422 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.420+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2664860766 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f62a402ff10 msgr2=0x7f62a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:16.422 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.420+0000 7f62bf3c9700 1 --2- 172.21.15.149:0/2664860766 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f62a402ff10 0x7f62a40323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f62b40096a0 tx=0x7f62b4006b40).stop 2024-02-12T16:41:16.422 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.420+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2664860766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b8060280 msgr2=0x7f62b80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:16.422 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.420+0000 7f62bf3c9700 1 --2- 172.21.15.149:0/2664860766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b8060280 0x7f62b80f6b70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f62a80089f0 tx=0x7f62a800b040).stop 2024-02-12T16:41:16.423 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.420+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2664860766 shutdown_connections 2024-02-12T16:41:16.423 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.420+0000 7f62bf3c9700 1 --2- 172.21.15.149:0/2664860766 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7f62a402ff10 0x7f62a40323d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:16.423 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.420+0000 7f62bf3c9700 1 --2- 172.21.15.149:0/2664860766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b8060280 0x7f62b80f6b70 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:16.423 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.420+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2664860766 >> 172.21.15.149:0/2664860766 conn(0x7f62b80f0b70 msgr2=0x7f62b80f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:16.423 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.420+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2664860766 shutdown_connections 2024-02-12T16:41:16.423 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:16.420+0000 7f62bf3c9700 1 -- 172.21.15.149:0/2664860766 wait complete. 2024-02-12T16:41:16.423 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:41:17.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:17 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:41:15.532057+0000 mgr.smithi104.ohwbhf (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:17.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:17 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:41:16.419465+0000 mon.smithi104 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.149:0/2664860766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:41:18.068 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:41:18.069 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:41:19.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:19 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:41:17.532392+0000 mgr.smithi104.ohwbhf (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:19.888 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.882+0000 7fb7d7063700 1 -- 172.21.15.149:0/2826201433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d00f78e0 msgr2=0x7fb7d00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:19.888 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.882+0000 7fb7d7063700 1 --2- 172.21.15.149:0/2826201433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d00f78e0 0x7fb7d00f7d00 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fb7c4009fb0 tx=0x7fb7c4009b30).stop 2024-02-12T16:41:19.888 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.882+0000 7fb7d7063700 1 -- 172.21.15.149:0/2826201433 shutdown_connections 2024-02-12T16:41:19.888 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.882+0000 7fb7d7063700 1 --2- 172.21.15.149:0/2826201433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d00f78e0 0x7fb7d00f7d00 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:19.888 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.882+0000 7fb7d7063700 1 -- 172.21.15.149:0/2826201433 >> 172.21.15.149:0/2826201433 conn(0x7fb7d00f0120 msgr2=0x7fb7d00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:19.888 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.882+0000 7fb7d7063700 1 -- 172.21.15.149:0/2826201433 shutdown_connections 2024-02-12T16:41:19.888 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.882+0000 7fb7d7063700 1 -- 172.21.15.149:0/2826201433 wait complete. 2024-02-12T16:41:19.888 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.882+0000 7fb7d7063700 1 Processor -- start 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.882+0000 7fb7d7063700 1 -- start start 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.882+0000 7fb7d7063700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d00f78e0 0x7fb7d01065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.882+0000 7fb7d7063700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0106ad0 con 0x7fb7d00f78e0 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.883+0000 7fb7d6061700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d00f78e0 0x7fb7d01065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.883+0000 7fb7d6061700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d00f78e0 0x7fb7d01065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:60822/0 (socket says 172.21.15.149:60822) 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.883+0000 7fb7d6061700 1 -- 172.21.15.149:0/1844299695 learned_addr learned my addr 172.21.15.149:0/1844299695 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.883+0000 7fb7d6061700 1 -- 172.21.15.149:0/1844299695 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c4005040 con 0x7fb7d00f78e0 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.883+0000 7fb7d6061700 1 --2- 172.21.15.149:0/1844299695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d00f78e0 0x7fb7d01065e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fb7c40089f0 tx=0x7fb7c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.883+0000 7fb7c2ffd700 1 -- 172.21.15.149:0/1844299695 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c4009320 con 0x7fb7d00f78e0 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.883+0000 7fb7d7063700 1 -- 172.21.15.149:0/1844299695 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d0106cd0 con 0x7fb7d00f78e0 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.883+0000 7fb7c2ffd700 1 -- 172.21.15.149:0/1844299695 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7c4013380 con 0x7fb7d00f78e0 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.883+0000 7fb7c2ffd700 1 -- 172.21.15.149:0/1844299695 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c4014e20 con 0x7fb7d00f78e0 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.883+0000 7fb7d7063700 1 -- 172.21.15.149:0/1844299695 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d0107160 con 0x7fb7d00f78e0 2024-02-12T16:41:19.889 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.884+0000 7fb7c2ffd700 1 -- 172.21.15.149:0/1844299695 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7c40095f0 con 0x7fb7d00f78e0 2024-02-12T16:41:19.890 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.884+0000 7fb7d7063700 1 -- 172.21.15.149:0/1844299695 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d0045ad0 con 0x7fb7d00f78e0 2024-02-12T16:41:19.890 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.884+0000 7fb7c2ffd700 1 --2- 172.21.15.149:0/1844299695 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb7bc02ff10 0x7fb7bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:19.890 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.884+0000 7fb7c2ffd700 1 -- 172.21.15.149:0/1844299695 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb7c4040e90 con 0x7fb7d00f78e0 2024-02-12T16:41:19.890 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.884+0000 7fb7d5860700 1 --2- 172.21.15.149:0/1844299695 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb7bc02ff10 0x7fb7bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:19.890 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.885+0000 7fb7d5860700 1 --2- 172.21.15.149:0/1844299695 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb7bc02ff10 0x7fb7bc0323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb7cc0096a0 tx=0x7fb7cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:19.891 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:19.889+0000 7fb7c2ffd700 1 -- 172.21.15.149:0/1844299695 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7c4014420 con 0x7fb7d00f78e0 2024-02-12T16:41:20.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.092+0000 7fb7d7063700 1 -- 172.21.15.149:0/1844299695 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7d0108010 con 0x7fb7d00f78e0 2024-02-12T16:41:20.095 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.093+0000 7fb7c2ffd700 1 -- 172.21.15.149:0/1844299695 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb7c401d9d0 con 0x7fb7d00f78e0 2024-02-12T16:41:20.096 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:41:20.096 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:41:20.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.094+0000 7fb7d7063700 1 -- 172.21.15.149:0/1844299695 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb7bc02ff10 msgr2=0x7fb7bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:20.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.094+0000 7fb7d7063700 1 --2- 172.21.15.149:0/1844299695 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb7bc02ff10 0x7fb7bc0323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb7cc0096a0 tx=0x7fb7cc006b40).stop 2024-02-12T16:41:20.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.094+0000 7fb7d7063700 1 -- 172.21.15.149:0/1844299695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d00f78e0 msgr2=0x7fb7d01065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:20.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.095+0000 7fb7d7063700 1 --2- 172.21.15.149:0/1844299695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d00f78e0 0x7fb7d01065e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fb7c40089f0 tx=0x7fb7c400b040).stop 2024-02-12T16:41:20.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.095+0000 7fb7d7063700 1 -- 172.21.15.149:0/1844299695 shutdown_connections 2024-02-12T16:41:20.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.095+0000 7fb7d7063700 1 --2- 172.21.15.149:0/1844299695 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7fb7bc02ff10 0x7fb7bc0323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:20.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.095+0000 7fb7d7063700 1 --2- 172.21.15.149:0/1844299695 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb7d00f78e0 0x7fb7d01065e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:20.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.095+0000 7fb7d7063700 1 -- 172.21.15.149:0/1844299695 >> 172.21.15.149:0/1844299695 conn(0x7fb7d00f0120 msgr2=0x7fb7d00f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:20.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.095+0000 7fb7d7063700 1 -- 172.21.15.149:0/1844299695 shutdown_connections 2024-02-12T16:41:20.097 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:20.095+0000 7fb7d7063700 1 -- 172.21.15.149:0/1844299695 wait complete. 2024-02-12T16:41:20.098 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:41:20.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:20 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:41:20.094257+0000 mon.smithi104 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.149:0/1844299695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:41:21.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:21 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:41:19.532737+0000 mgr.smithi104.ohwbhf (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:21.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-12T16:41:21.719 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T16:41:23.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.661+0000 7ff85a578700 1 -- 172.21.15.149:0/1409047561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8540f6510 msgr2=0x7ff8540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:23.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.661+0000 7ff85a578700 1 --2- 172.21.15.149:0/1409047561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8540f6510 0x7ff8540f6930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7ff844009fb0 tx=0x7ff844009b30).stop 2024-02-12T16:41:23.663 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.661+0000 7ff85a578700 1 -- 172.21.15.149:0/1409047561 shutdown_connections 2024-02-12T16:41:23.664 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.661+0000 7ff85a578700 1 --2- 172.21.15.149:0/1409047561 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8540f6510 0x7ff8540f6930 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:23.664 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.661+0000 7ff85a578700 1 -- 172.21.15.149:0/1409047561 >> 172.21.15.149:0/1409047561 conn(0x7ff8540f1c70 msgr2=0x7ff8540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:23.664 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.661+0000 7ff85a578700 1 -- 172.21.15.149:0/1409047561 shutdown_connections 2024-02-12T16:41:23.664 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.661+0000 7ff85a578700 1 -- 172.21.15.149:0/1409047561 wait complete. 2024-02-12T16:41:23.665 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.661+0000 7ff85a578700 1 Processor -- start 2024-02-12T16:41:23.665 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.662+0000 7ff85a578700 1 -- start start 2024-02-12T16:41:23.665 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.662+0000 7ff85a578700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8540f6510 0x7ff8541067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:23.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.662+0000 7ff85a578700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff854106cd0 con 0x7ff8540f6510 2024-02-12T16:41:23.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.662+0000 7ff853fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8540f6510 0x7ff8541067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:23.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.662+0000 7ff853fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8540f6510 0x7ff8541067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.149:60844/0 (socket says 172.21.15.149:60844) 2024-02-12T16:41:23.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.662+0000 7ff853fff700 1 -- 172.21.15.149:0/3209761231 learned_addr learned my addr 172.21.15.149:0/3209761231 (peer_addr_for_me v2:172.21.15.149:0/0) 2024-02-12T16:41:23.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.662+0000 7ff853fff700 1 -- 172.21.15.149:0/3209761231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff844005040 con 0x7ff8540f6510 2024-02-12T16:41:23.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.663+0000 7ff853fff700 1 --2- 172.21.15.149:0/3209761231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8540f6510 0x7ff8541067e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7ff844005010 tx=0x7ff84400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:23.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.663+0000 7ff8517fa700 1 -- 172.21.15.149:0/3209761231 <== mon.0 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff844009320 con 0x7ff8540f6510 2024-02-12T16:41:23.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.663+0000 7ff8517fa700 1 -- 172.21.15.149:0/3209761231 <== mon.0 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff844013380 con 0x7ff8540f6510 2024-02-12T16:41:23.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.663+0000 7ff85a578700 1 -- 172.21.15.149:0/3209761231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff854106ed0 con 0x7ff8540f6510 2024-02-12T16:41:23.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.663+0000 7ff8517fa700 1 -- 172.21.15.149:0/3209761231 <== mon.0 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff844014e50 con 0x7ff8540f6510 2024-02-12T16:41:23.666 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.663+0000 7ff85a578700 1 -- 172.21.15.149:0/3209761231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff854107360 con 0x7ff8540f6510 2024-02-12T16:41:23.667 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.664+0000 7ff8517fa700 1 -- 172.21.15.149:0/3209761231 <== mon.0 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8440095f0 con 0x7ff8540f6510 2024-02-12T16:41:23.667 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.664+0000 7ff85a578700 1 -- 172.21.15.149:0/3209761231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff854045ad0 con 0x7ff8540f6510 2024-02-12T16:41:23.667 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.664+0000 7ff8517fa700 1 --2- 172.21.15.149:0/3209761231 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff84002ff10 0x7ff8400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-12T16:41:23.667 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.664+0000 7ff8517fa700 1 -- 172.21.15.149:0/3209761231 <== mon.0 v2:172.21.15.104:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff844040e60 con 0x7ff8540f6510 2024-02-12T16:41:23.667 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.664+0000 7ff84bfff700 1 --2- 172.21.15.149:0/3209761231 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff84002ff10 0x7ff8400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-12T16:41:23.667 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.665+0000 7ff84bfff700 1 --2- 172.21.15.149:0/3209761231 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff84002ff10 0x7ff8400323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7ff83c0096a0 tx=0x7ff83c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-12T16:41:23.671 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.669+0000 7ff8517fa700 1 -- 172.21.15.149:0/3209761231 <== mon.0 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff844014420 con 0x7ff8540f6510 2024-02-12T16:41:23.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:23 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: cluster 2024-02-12T16:41:21.533139+0000 mgr.smithi104.ohwbhf (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T16:41:23.877 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.874+0000 7ff85a578700 1 -- 172.21.15.149:0/3209761231 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff854108210 con 0x7ff8540f6510 2024-02-12T16:41:23.877 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.875+0000 7ff8517fa700 1 -- 172.21.15.149:0/3209761231 <== mon.0 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff84401d980 con 0x7ff8540f6510 2024-02-12T16:41:23.877 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-02-12T16:41:23.877 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"aa0fd796-c9c3-11ee-95b9-87774f69a715","modified":"2024-02-12T16:28:18.956483Z","created":"2024-02-12T16:28:18.956483Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T16:41:23.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.876+0000 7ff85a578700 1 -- 172.21.15.149:0/3209761231 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff84002ff10 msgr2=0x7ff8400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:23.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.876+0000 7ff85a578700 1 --2- 172.21.15.149:0/3209761231 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff84002ff10 0x7ff8400323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7ff83c0096a0 tx=0x7ff83c006b40).stop 2024-02-12T16:41:23.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.876+0000 7ff85a578700 1 -- 172.21.15.149:0/3209761231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8540f6510 msgr2=0x7ff8541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-12T16:41:23.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.876+0000 7ff85a578700 1 --2- 172.21.15.149:0/3209761231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8540f6510 0x7ff8541067e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7ff844005010 tx=0x7ff84400b040).stop 2024-02-12T16:41:23.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.876+0000 7ff85a578700 1 -- 172.21.15.149:0/3209761231 shutdown_connections 2024-02-12T16:41:23.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.876+0000 7ff85a578700 1 --2- 172.21.15.149:0/3209761231 >> [v2:172.21.15.104:6800/4169661527,v1:172.21.15.104:6801/4169661527] conn(0x7ff84002ff10 0x7ff8400323d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:23.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.876+0000 7ff85a578700 1 --2- 172.21.15.149:0/3209761231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8540f6510 0x7ff8541067e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-12T16:41:23.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.876+0000 7ff85a578700 1 -- 172.21.15.149:0/3209761231 >> 172.21.15.149:0/3209761231 conn(0x7ff8540f1c70 msgr2=0x7ff8540f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-12T16:41:23.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.876+0000 7ff85a578700 1 -- 172.21.15.149:0/3209761231 shutdown_connections 2024-02-12T16:41:23.879 INFO:teuthology.orchestra.run.smithi149.stderr:2024-02-12T16:41:23.877+0000 7ff85a578700 1 -- 172.21.15.149:0/3209761231 wait complete. 2024-02-12T16:41:23.880 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-02-12T16:41:24.520 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-12T16:41:24.522 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-12T16:41:24.522 DEBUG:teuthology.orchestra.run.smithi104:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-12T16:41:24.546 DEBUG:teuthology.orchestra.run.smithi149:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-12T16:41:24.564 INFO:tasks.cephadm:Stopping all daemons... 2024-02-12T16:41:24.565 INFO:tasks.cephadm.mon.smithi104:Stopping mon.smithi104... 2024-02-12T16:41:24.565 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104 2024-02-12T16:41:24.616 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: audit 2024-02-12T16:41:23.876110+0000 mon.smithi104 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.149:0/3209761231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:41:24.926 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:24 smithi104 systemd[1]: Stopping Ceph mon.smithi104 for aa0fd796-c9c3-11ee-95b9-87774f69a715... 2024-02-12T16:41:24.926 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:41:24.762+0000 7ffbef0f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi104 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-12T16:41:24.926 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:24 smithi104 ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104[39424]: debug 2024-02-12T16:41:24.762+0000 7ffbef0f5700 -1 mon.smithi104@0(leader) e1 *** Got Signal Terminated *** 2024-02-12T16:41:25.628 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 12 16:41:25 smithi104 podman[45326]: ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715-mon.smithi104 2024-02-12T16:41:25.632 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi104.service' 2024-02-12T16:41:25.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:41:25.736 INFO:tasks.cephadm.mon.smithi104:Stopped mon.smithi104 2024-02-12T16:41:25.737 INFO:tasks.cephadm.mon.smithi149:Stopping mon.smithi149... 2024-02-12T16:41:25.737 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi149 2024-02-12T16:41:25.774 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi149.service: Unit ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi149.service not loaded. 2024-02-12T16:41:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-12T16:41:25.777 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi149" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi149 with status 5: 'sudo systemctl stop ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi149' 2024-02-12T16:41:25.778 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-12T16:41:25.779 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688/remote/ubuntu@smithi104.front.sepia.ceph.com/crash 2024-02-12T16:41:25.780 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/crash -- . 2024-02-12T16:41:25.841 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688/remote/ubuntu@smithi149.front.sepia.ceph.com/crash 2024-02-12T16:41:25.842 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/crash -- . 2024-02-12T16:41:25.879 INFO:teuthology.orchestra.run.smithi149.stderr:tar: /var/lib/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-12T16:41:25.879 INFO:teuthology.orchestra.run.smithi149.stderr:tar: Error is not recoverable: exiting now 2024-02-12T16:41:25.880 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-12T16:41:25.880 DEBUG:teuthology.orchestra.run.smithi104:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-12T16:41:25.915 INFO:tasks.cephadm:Compressing logs... 2024-02-12T16:41:25.915 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-12T16:41:25.957 DEBUG:teuthology.orchestra.run.smithi149:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-12T16:41:25.984 INFO:teuthology.orchestra.run.smithi104.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-12T16:41:25.984 INFO:teuthology.orchestra.run.smithi104.stderr:: No such file or directory 2024-02-12T16:41:25.984 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-12T16:41:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-12T16:41:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-12T16:41:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-volume.log: /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-mon.smithi104.log 2024-02-12T16:41:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-mgr.smithi104.ohwbhf.log 2024-02-12T16:41:25.986 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-12T16:41:25.986 INFO:teuthology.orchestra.run.smithi104.stderr: 89.0% -- replaced with /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-12T16:41:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-mon.smithi104.log: 89.5% -- replaced with /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-12T16:41:25.986 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph.log 2024-02-12T16:41:25.987 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-mgr.smithi104.ohwbhf.log: /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph.log: 88.9% -- replaced with /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-12T16:41:25.987 INFO:teuthology.orchestra.run.smithi149.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-12T16:41:25.987 INFO:teuthology.orchestra.run.smithi149.stderr:: No such file or directory 2024-02-12T16:41:25.987 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-12T16:41:25.988 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-12T16:41:25.989 INFO:teuthology.orchestra.run.smithi149.stderr: 95.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-12T16:41:25.989 INFO:teuthology.orchestra.run.smithi104.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-12T16:41:25.991 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-02-12T16:41:25.991 INFO:teuthology.orchestra.run.smithi149.stderr:real 0m0.014s 2024-02-12T16:41:25.991 INFO:teuthology.orchestra.run.smithi149.stderr:user 0m0.010s 2024-02-12T16:41:25.991 INFO:teuthology.orchestra.run.smithi149.stderr:sys 0m0.015s 2024-02-12T16:41:26.080 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-mgr.smithi104.ohwbhf.log: file size changed while zipping 2024-02-12T16:41:26.080 INFO:teuthology.orchestra.run.smithi104.stderr: 92.8% -- replaced with /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-mgr.smithi104.ohwbhf.log.gz 2024-02-12T16:41:26.155 INFO:teuthology.orchestra.run.smithi104.stderr: 90.7% -- replaced with /var/log/ceph/aa0fd796-c9c3-11ee-95b9-87774f69a715/ceph-mon.smithi104.log.gz 2024-02-12T16:41:26.157 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-02-12T16:41:26.157 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m0.184s 2024-02-12T16:41:26.158 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m0.273s 2024-02-12T16:41:26.158 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.024s 2024-02-12T16:41:26.158 INFO:tasks.cephadm:Archiving logs... 2024-02-12T16:41:26.158 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688/remote/ubuntu@smithi104.front.sepia.ceph.com/log 2024-02-12T16:41:26.159 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-12T16:41:26.284 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688/remote/ubuntu@smithi149.front.sepia.ceph.com/log 2024-02-12T16:41:26.285 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-12T16:41:26.318 INFO:tasks.cephadm:Removing cluster... 2024-02-12T16:41:26.318 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 --force 2024-02-12T16:45:08.174 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid aa0fd796-c9c3-11ee-95b9-87774f69a715 --force 2024-02-12T16:45:08.479 INFO:tasks.cephadm:Removing cephadm ... 2024-02-12T16:45:08.479 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-12T16:45:08.499 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-12T16:45:08.517 INFO:tasks.cephadm:Teardown complete 2024-02-12T16:45:08.517 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi149 with status 5: 'sudo systemctl stop ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi149' 2024-02-12T16:45:08.743 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cda98796b93b4789b17cb2f774fc06e7 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi149 with status 5: 'sudo systemctl stop ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi149' 2024-02-12T16:45:08.748 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-12T16:45:08.757 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-12T16:45:08.771 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-12T16:45:08.771 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-12T16:45:08.787 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-12T16:45:08.866 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-12T16:45:08.875 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-12T16:45:08.876 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T16:45:08.878 DEBUG:teuthology.orchestra.run.smithi149:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T16:45:08.894 INFO:teuthology.orchestra.run.smithi104.stderr:bash: ntpq: command not found 2024-02-12T16:45:08.895 INFO:teuthology.orchestra.run.smithi149.stderr:bash: ntpq: command not found 2024-02-12T16:45:08.898 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-12T16:45:08.898 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-02-12T16:45:08.898 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 6 -1527us[-1547us] +/- 93ms 2024-02-12T16:45:08.898 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 +62us[ +43us] +/- 64ms 2024-02-12T16:45:08.898 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +1158us[+1139us] +/- 69ms 2024-02-12T16:45:08.898 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:45:08.900 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-12T16:45:08.900 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-02-12T16:45:08.900 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 -1548us[-1548us] +/- 93ms 2024-02-12T16:45:08.901 INFO:teuthology.orchestra.run.smithi149.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 +54us[ +50us] +/- 64ms 2024-02-12T16:45:08.901 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +1178us[+1178us] +/- 69ms 2024-02-12T16:45:08.901 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-12T16:45:08.901 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-12T16:45:08.910 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-12T16:45:08.911 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-12T16:45:08.932 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-12T16:45:09.003 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-12T16:45:09.067 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-12T16:45:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:09.097 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-12T16:45:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:09.126 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-12T16:45:09.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-12T16:45:09.145 INFO:teuthology.task.internal:Duration was 1589.513338 seconds 2024-02-12T16:45:09.146 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-12T16:45:09.155 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-12T16:45:09.155 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-12T16:45:09.157 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-12T16:45:09.200 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-12T16:45:09.212 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-12T16:45:09.600 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-12T16:45:09.601 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-02-12T16:45:09.601 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-12T16:45:09.624 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi149.front.sepia.ceph.com 2024-02-12T16:45:09.625 DEBUG:teuthology.orchestra.run.smithi149:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-12T16:45:09.647 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-12T16:45:09.647 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-12T16:45:09.667 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-12T16:45:09.736 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-12T16:45:09.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-12T16:45:09.739 DEBUG:teuthology.orchestra.run.smithi149:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-12T16:45:09.830 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-12T16:45:09.841 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-12T16:45:09.841 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-12T16:45:09.868 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-12T16:45:09.896 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-12T16:45:09.906 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-12T16:45:09.911 DEBUG:teuthology.orchestra.run.smithi149:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-12T16:45:09.935 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-02-12T16:45:09.962 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = core 2024-02-12T16:45:09.982 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-12T16:45:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:10.020 DEBUG:teuthology.orchestra.run.smithi149:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-12T16:45:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:10.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-12T16:45:10.049 INFO:teuthology.task.internal:Transferring archived files... 2024-02-12T16:45:10.050 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688/remote/smithi104 2024-02-12T16:45:10.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-12T16:45:10.100 DEBUG:teuthology.misc:Transferring archived files from smithi149:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688/remote/smithi149 2024-02-12T16:45:10.101 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-12T16:45:10.139 INFO:teuthology.task.internal:Removing archive directory... 2024-02-12T16:45:10.139 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-12T16:45:10.142 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-12T16:45:10.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-12T16:45:10.202 INFO:teuthology.task.internal:Not uploading archives. 2024-02-12T16:45:10.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-12T16:45:10.212 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-12T16:45:10.212 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-12T16:45:10.214 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-12T16:45:10.229 INFO:teuthology.orchestra.run.smithi104.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 16:45 /home/ubuntu/cephtest 2024-02-12T16:45:10.248 INFO:teuthology.orchestra.run.smithi149.stdout: 526076 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 16:45 /home/ubuntu/cephtest 2024-02-12T16:45:10.250 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-12T16:45:10.260 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-12T16:45:10.280 INFO:teuthology.nuke:Checking targets against current locks 2024-02-12T16:45:10.308 DEBUG:teuthology.nuke:shortname: smithi104 2024-02-12T16:45:10.309 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-12T16:45:10.334 DEBUG:teuthology.nuke:shortname: smithi149 2024-02-12T16:45:10.335 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-12T16:45:10.338 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 16:09:41.952892', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBrJUGIEQIwpx46m3RT+Ns3h9SYOxQsiDQZ1EmpPCSVBVxHR+WyyvvHsZqyhn1LTg9m4vWwBRcd7Pl1fB8IiJus='} 2024-02-12T16:45:10.361 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556688', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 16:09:41.951384', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkcYX3D8sRb4OJbwChslFMKsi0ay2UAfVox0OqzpNCtXFzO0bxiVpTPp9/w9nTE8nrRKnDCJLVzcHKqmfb6ZU8='} 2024-02-12T16:45:10.389 INFO:teuthology.orchestra.console.smithi104:Power off 2024-02-12T16:45:10.389 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-12T16:45:10.415 DEBUG:teuthology.orchestra.console.smithi104:power off output: Chassis Power Control: Down/Off 2024-02-12T16:45:10.416 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T16:45:10.431 INFO:teuthology.orchestra.console.smithi149:Power off 2024-02-12T16:45:10.431 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-12T16:45:10.441 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-02-12T16:45:10.453 DEBUG:teuthology.orchestra.console.smithi149:power off output: Chassis Power Control: Down/Off 2024-02-12T16:45:10.454 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T16:45:10.479 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-02-12T16:45:14.442 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T16:45:14.480 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T16:45:14.552 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-02-12T16:45:14.592 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-02-12T16:45:18.556 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T16:45:18.593 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T16:45:18.666 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off 2024-02-12T16:45:18.703 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2024-02-12T16:45:18.767 INFO:teuthology.orchestra.console.smithi104:Power off completed 2024-02-12T16:45:18.804 INFO:teuthology.orchestra.console.smithi149:Power off completed 2024-02-12T16:45:18.909 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1589.5133383274078 failure_reason: 'Command failed on smithi149 with status 5: ''sudo systemctl stop ceph-aa0fd796-c9c3-11ee-95b9-87774f69a715@mon.smithi149''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cda98796b93b4789b17cb2f774fc06e7 status: fail success: false 2024-02-12T16:45:18.909 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-12T16:45:18.986 INFO:teuthology.run:FAIL