2024-02-19T21:35:33.398 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T21:35:33.398 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781 -- /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781/orig.config.yaml 2024-02-19T21:35:33.444 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T21:35:33.533 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781 branch: pacific-release description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7566781' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: pacific-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - Degraded data redundancy - \(PG_ - MDS_INSUFFICIENT_STANDBY - deprecated feature inline_data - compat changed unexpectedly - overall HEALTH_ - OSD_DOWN - OSD_ - but it is still running - is not responding sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: pacific-release sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 264 sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: pacific-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 targets: smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHyUUq7G/aCUWjKEixnVGNXk6NckngxCGMXn1sh+yFhF3Ccv6o5Nxpbw9rtT4QGa0y5SS7r/U4TdDejajeTuwsk= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEutp+TVxGTaagfKrGqAAJK5kF2QB/T19msqiiot6i+abtSbFTMGkxg2zUwQ++Z/Rhk4Srl20LgtjHD9EMhrqFM= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_19:25:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-19T21:35:33.533 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa; will attempt to use it 2024-02-19T21:35:33.534 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks 2024-02-19T21:35:33.534 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T21:35:33.536 INFO:teuthology.task.internal:Checking packages... 2024-02-19T21:35:33.559 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '81bd20d634209c7cb82c18be12b4b5a05643ebf1' 2024-02-19T21:35:33.560 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T21:35:33.560 INFO:teuthology.packaging:ref: None 2024-02-19T21:35:33.560 INFO:teuthology.packaging:tag: None 2024-02-19T21:35:33.560 INFO:teuthology.packaging:branch: pacific-release 2024-02-19T21:35:33.560 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:35:33.560 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific-release 2024-02-19T21:35:33.774 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-548.g81bd20d6 2024-02-19T21:35:33.775 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T21:35:33.782 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T21:35:33.783 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T21:35:33.796 INFO:teuthology.task.internal:Saving configuration 2024-02-19T21:35:33.815 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T21:35:33.821 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T21:35:33.843 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:30:34.437686', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHyUUq7G/aCUWjKEixnVGNXk6NckngxCGMXn1sh+yFhF3Ccv6o5Nxpbw9rtT4QGa0y5SS7r/U4TdDejajeTuwsk='} 2024-02-19T21:35:33.869 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:30:34.439147', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEutp+TVxGTaagfKrGqAAJK5kF2QB/T19msqiiot6i+abtSbFTMGkxg2zUwQ++Z/Rhk4Srl20LgtjHD9EMhrqFM='} 2024-02-19T21:35:33.869 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T21:35:33.875 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-19T21:35:33.875 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-19T21:35:33.875 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T21:35:33.959 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f923a4de0d0>, signals=[15]) 2024-02-19T21:35:33.959 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T21:35:33.972 INFO:teuthology.task.internal:Opening connections... 2024-02-19T21:35:33.972 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2024-02-19T21:35:33.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:35:34.049 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-02-19T21:35:34.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:35:34.120 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T21:35:34.127 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2024-02-19T21:35:34.150 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2024-02-19T21:35:34.150 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2024-02-19T21:35:34.211 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream" 2024-02-19T21:35:34.211 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="8" 2024-02-19T21:35:34.211 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos" 2024-02-19T21:35:34.211 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:35:34.211 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="8" 2024-02-19T21:35:34.211 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:35:34.211 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:35:34.211 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2024-02-19T21:35:34.211 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:35:34.211 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:35:34.212 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:35:34.212 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:35:34.212 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:35:34.212 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2024-02-19T21:35:34.238 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-02-19T21:35:34.260 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-02-19T21:35:34.261 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="8" 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="8" 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:35:34.322 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:35:34.323 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:35:34.323 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-02-19T21:35:34.338 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T21:35:34.348 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T21:35:34.388 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T21:35:34.388 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:35:34.390 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:35:34.410 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T21:35:34.417 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T21:35:34.417 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:35:34.451 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:35:34.483 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T21:35:34.536 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T21:35:34.551 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-19T21:35:34.551 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-19T21:35:34.551 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:35:34.551 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:35:34.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:35:34.572 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-02-19T21:35:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:35:34.573 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-02-19T21:35:34.633 INFO:teuthology.orchestra.run.smithi143.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:35:34.633 INFO:teuthology.task.kernel:Running kernel on smithi143: 4.18.0-489.el8.x86_64 2024-02-19T21:35:34.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum install -y kernel 2024-02-19T21:35:34.634 INFO:teuthology.orchestra.run.smithi173.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:35:34.634 INFO:teuthology.task.kernel:Running kernel on smithi173: 4.18.0-489.el8.x86_64 2024-02-19T21:35:34.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-02-19T21:35:52.559 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - AppStream 1.8 MB/s | 27 MB 00:15 2024-02-19T21:35:53.178 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 8 - AppStream 1.7 MB/s | 27 MB 00:15 2024-02-19T21:35:58.059 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 10 MB 00:00 2024-02-19T21:35:58.579 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 10 MB 00:00 2024-02-19T21:35:59.935 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras 63 kB/s | 18 kB 00:00 2024-02-19T21:36:00.446 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 8 - Extras 65 kB/s | 18 kB 00:00 2024-02-19T21:36:01.527 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras common packages 4.7 kB/s | 6.9 kB 00:01 2024-02-19T21:36:01.803 INFO:teuthology.orchestra.run.smithi173.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-02-19T21:36:01.896 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 8 - Extras common packages 5.2 kB/s | 6.9 kB 00:01 2024-02-19T21:36:02.181 INFO:teuthology.orchestra.run.smithi143.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-02-19T21:36:04.114 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 16 MB 00:02 2024-02-19T21:36:07.829 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 450 kB/s | 24 kB 00:00 2024-02-19T21:36:10.767 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:36:10.768 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:36:10.883 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:36:10.884 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:36:10.884 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-19T21:36:10.884 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:36:10.884 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:36:10.884 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:36:10.884 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-19T21:36:10.884 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:36:10.885 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:36:10.885 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:36:10.885 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:36:10.885 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:36:10.885 INFO:teuthology.orchestra.run.smithi173.stdout:Install 3 Packages 2024-02-19T21:36:10.885 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:36:10.885 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 90 M 2024-02-19T21:36:10.885 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 97 M 2024-02-19T21:36:10.886 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:36:12.505 INFO:teuthology.orchestra.run.smithi173.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 7.2 MB/s | 10 MB 00:01 2024-02-19T21:36:14.572 INFO:teuthology.orchestra.run.smithi173.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 10 MB/s | 36 MB 00:03 2024-02-19T21:36:15.247 INFO:teuthology.orchestra.run.smithi173.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 10 MB/s | 43 MB 00:04 2024-02-19T21:36:15.248 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:36:15.248 INFO:teuthology.orchestra.run.smithi173.stdout:Total 21 MB/s | 90 MB 00:04 2024-02-19T21:36:15.930 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:36:16.344 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:36:16.344 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:36:17.583 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:36:17.607 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:36:20.349 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:36:20.841 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:36:24.445 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:36:24.874 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:36:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:36:28.639 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:36:50.495 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:36:50.518 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:36:50.879 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:36:50.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:36:50.879 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:36:51.387 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:36:51.387 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:36:51.387 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:36:51.387 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T21:36:51.387 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T21:36:51.387 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:36:51.387 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:36:51.548 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:36:51.765 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:36:51.765 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:36:51.765 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T21:36:51.765 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T21:36:51.766 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T21:36:51.766 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-19T21:36:51.766 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T21:36:51.766 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:36:51.977 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:36:51.977 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:36:51.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-02-19T21:36:52.011 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-02-19T21:36:52.011 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-02-19T21:36:52.011 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.02 2024-02-19T21:36:52.011 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 129.el8 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Group : System Environment/Base 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 9265586 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Relocations : (not relocatable) 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : CentOS BuildSystem 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T21:36:52.012 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-02-19T21:36:52.013 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-02-19T21:36:52.013 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:36:52.013 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T21:36:52.013 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T21:36:52.013 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T21:36:52.013 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-02-19T21:36:52.013 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:36:52.013 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T21:36:52.014 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T21:36:52.014 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 4.18.0-539.el8.x86_64 2024-02-19T21:36:52.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T21:36:53.801 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-02-19T21:36:55.654 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-02-19T21:36:55.656 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-02-19T21:36:55.699 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.9htesNhf6Q 2024-02-19T21:36:55.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9htesNhf6Q 2024-02-19T21:36:55.773 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.9htesNhf6Q 2024-02-19T21:36:55.936 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.9htesNhf6Q is 6KB 2024-02-19T21:36:55.984 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.9htesNhf6Q 2024-02-19T21:36:56.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T21:36:56.073 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T21:36:56.073 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T21:36:56.073 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T21:36:56.073 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T21:36:56.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T21:36:56.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-02-19T21:37:26.243 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-19T21:37:26.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-19T21:37:26.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:38:00.892 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-19T21:38:09.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-19T21:38:09.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:38:28.284 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-02-19T21:38:40.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-19T21:38:40.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:38:40.700 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-02-19T21:38:41.069 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-19T21:38:41.070 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-539.el8.x86_64"... 2024-02-19T21:38:41.070 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-02-19T21:38:41.087 INFO:teuthology.orchestra.run.smithi173.stdout:4.18.0-539.el8.x86_64 2024-02-19T21:38:41.087 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T21:38:41.087 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T21:38:41.087 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T21:38:42.088 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-19T21:38:42.088 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T21:38:42.193 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-02-19T21:38:42.205 DEBUG:teuthology.parallel:result is None 2024-02-19T21:40:27.384 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 62 kB/s | 16 MB 04:25 2024-02-19T21:40:31.126 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 386 kB/s | 24 kB 00:00 2024-02-19T21:40:34.064 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:40:34.064 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:40:34.186 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:40:34.187 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:40:34.187 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-02-19T21:40:34.187 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:40:34.187 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:40:34.187 INFO:teuthology.orchestra.run.smithi143.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:40:34.187 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-02-19T21:40:34.187 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:40:34.187 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:40:34.188 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:40:34.188 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:40:34.188 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:40:34.188 INFO:teuthology.orchestra.run.smithi143.stdout:Install 3 Packages 2024-02-19T21:40:34.188 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:40:34.188 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 90 M 2024-02-19T21:40:34.189 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 97 M 2024-02-19T21:40:34.189 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:40:35.011 INFO:teuthology.orchestra.run.smithi143.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 16 MB/s | 10 MB 00:00 2024-02-19T21:40:36.169 INFO:teuthology.orchestra.run.smithi143.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 20 MB/s | 36 MB 00:01 2024-02-19T21:40:36.686 INFO:teuthology.orchestra.run.smithi143.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 19 MB/s | 43 MB 00:02 2024-02-19T21:40:36.686 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:40:36.686 INFO:teuthology.orchestra.run.smithi143.stdout:Total 36 MB/s | 90 MB 00:02 2024-02-19T21:40:37.365 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:40:37.777 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:40:37.777 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:40:39.011 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:40:39.039 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:40:41.814 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:40:42.320 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:40:45.909 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:40:46.303 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:40:49.741 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:40:50.037 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:41:12.434 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:41:12.459 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:41:12.805 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:41:12.805 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:41:12.805 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:41:13.798 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:41:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:41:13.799 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:41:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T21:41:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T21:41:13.799 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:41:13.799 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:41:13.981 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:41:14.204 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:41:14.204 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:41:14.205 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T21:41:14.205 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T21:41:14.205 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T21:41:14.205 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-19T21:41:14.205 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi143.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T21:41:14.205 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:41:14.415 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:41:14.416 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:41:14.416 DEBUG:teuthology.orchestra.run.smithi143:> sudo rpm -qi grub2-tools 2024-02-19T21:41:14.449 INFO:teuthology.orchestra.run.smithi143.stdout:Name : grub2-tools 2024-02-19T21:41:14.449 INFO:teuthology.orchestra.run.smithi143.stdout:Epoch : 1 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Version : 2.02 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Release : 129.el8 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: x86_64 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Group : System Environment/Base 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Size : 9265586 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:License : GPLv3+ 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Relocations : (not relocatable) 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Packager : CentOS BuildSystem 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:Vendor : CentOS 2024-02-19T21:41:14.450 INFO:teuthology.orchestra.run.smithi143.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T21:41:14.451 INFO:teuthology.orchestra.run.smithi143.stdout:Summary : Support tools for GRUB. 2024-02-19T21:41:14.451 INFO:teuthology.orchestra.run.smithi143.stdout:Description : 2024-02-19T21:41:14.451 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:41:14.451 INFO:teuthology.orchestra.run.smithi143.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T21:41:14.451 INFO:teuthology.orchestra.run.smithi143.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T21:41:14.451 INFO:teuthology.orchestra.run.smithi143.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T21:41:14.451 INFO:teuthology.orchestra.run.smithi143.stdout:hardware devices. 2024-02-19T21:41:14.451 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:41:14.451 INFO:teuthology.orchestra.run.smithi143.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T21:41:14.452 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T21:41:14.452 INFO:teuthology.task.kernel:Updating grub on smithi143 to boot 4.18.0-539.el8.x86_64 2024-02-19T21:41:14.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T21:41:15.954 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2024-02-19T21:41:18.181 INFO:teuthology.orchestra.run.smithi143.stderr:done 2024-02-19T21:41:18.183 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-02-19T21:41:18.223 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.XPb58uK1pv 2024-02-19T21:41:18.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XPb58uK1pv 2024-02-19T21:41:18.307 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.XPb58uK1pv 2024-02-19T21:41:18.452 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.XPb58uK1pv is 6KB 2024-02-19T21:41:18.504 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.XPb58uK1pv 2024-02-19T21:41:18.522 DEBUG:teuthology.orchestra.run.smithi143:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T21:41:18.589 INFO:teuthology.orchestra.run.smithi143.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T21:41:18.589 INFO:teuthology.orchestra.run.smithi143.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T21:41:18.589 INFO:teuthology.orchestra.run.smithi143.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T21:41:18.589 INFO:teuthology.orchestra.run.smithi143.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T21:41:18.590 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T21:41:18.763 DEBUG:teuthology.orchestra.run.smithi143:> sudo shutdown -r now 2024-02-19T21:41:48.766 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-19T21:41:48.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-02-19T21:41:48.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:42:48.771 DEBUG:teuthology.orchestra.remote:timed out 2024-02-19T21:42:57.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-02-19T21:42:57.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:42:58.094 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-02-19T21:42:58.441 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-02-19T21:42:58.441 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-539.el8.x86_64"... 2024-02-19T21:42:58.441 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-02-19T21:42:58.457 INFO:teuthology.orchestra.run.smithi143.stdout:4.18.0-539.el8.x86_64 2024-02-19T21:42:58.458 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T21:42:58.458 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T21:42:58.458 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T21:42:59.459 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-19T21:42:59.459 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T21:42:59.547 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2024-02-19T21:42:59.564 DEBUG:teuthology.parallel:result is None 2024-02-19T21:42:59.565 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T21:42:59.572 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T21:42:59.572 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T21:42:59.575 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T21:42:59.599 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T21:42:59.608 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T21:42:59.641 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T21:42:59.641 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T21:42:59.643 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T21:42:59.677 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T21:42:59.685 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T21:42:59.685 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T21:42:59.699 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T21:42:59.735 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:42:59.745 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:42:59.750 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:42:59.761 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:42:59.762 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T21:42:59.769 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T21:42:59.770 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T21:42:59.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T21:42:59.831 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T21:42:59.841 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T21:42:59.841 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T21:42:59.850 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T21:42:59.890 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:42:59.949 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:42:59.996 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:43:00.076 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:43:00.117 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-02-19T21:43:00.117 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T21:43:00.180 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:43:00.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:43:00.285 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:43:00.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:43:00.414 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-19T21:43:00.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T21:43:00.482 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2024-02-19T21:43:00.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-02-19T21:43:00.547 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:43:00.605 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:43:00.950 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T21:43:00.958 INFO:teuthology.task.internal:Starting timer... 2024-02-19T21:43:00.959 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T21:43:01.075 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T21:43:01.112 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-19T21:43:01.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo service auditd rotate 2024-02-19T21:43:01.212 INFO:teuthology.orchestra.run.smithi143.stdout:Rotating logs: 2024-02-19T21:43:01.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-02-19T21:43:01.292 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-02-19T21:43:01.294 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T21:43:01.294 DEBUG:teuthology.orchestra.run.smithi143:> /usr/sbin/getenforce 2024-02-19T21:43:01.332 INFO:teuthology.orchestra.run.smithi143.stdout:Permissive 2024-02-19T21:43:01.335 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-02-19T21:43:01.364 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-02-19T21:43:01.364 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi143.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-02-19T21:43:01.364 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:43:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:43:01.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:43:01.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:43:01.431 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T21:43:01.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo /usr/sbin/setenforce permissive 2024-02-19T21:43:01.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-02-19T21:43:01.732 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T21:43:01.744 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T21:43:01.762 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T21:43:01.763 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi143.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T21:49:01.515 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi143.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-02-19T21:49:01.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-02-19T21:49:01.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:49:01.593 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-02-19T21:49:01.662 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-02-19T21:49:01.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-19T21:49:01.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:49:01.732 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-02-19T21:49:01.801 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-02-19T21:49:01.802 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T21:49:01.813 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T21:49:01.813 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T21:49:01.814 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T21:49:01.816 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T21:49:01.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T21:49:01.846 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T21:49:01.860 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T21:49:01.872 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T21:49:01.886 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T21:49:01.928 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: ntpd: command not found 2024-02-19T21:49:01.939 INFO:teuthology.orchestra.run.smithi143.stdout:506 Cannot talk to daemon 2024-02-19T21:49:01.953 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T21:49:01.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-02-19T21:49:01.966 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T21:49:01.969 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-02-19T21:49:01.983 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T21:49:01.996 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T21:49:02.038 INFO:teuthology.orchestra.run.smithi143.stderr:bash: ntpq: command not found 2024-02-19T21:49:02.041 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T21:49:02.041 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-02-19T21:49:02.041 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:49:02.042 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:49:02.042 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:49:02.042 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:49:02.062 INFO:teuthology.orchestra.run.smithi173.stderr:bash: ntpq: command not found 2024-02-19T21:49:02.065 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T21:49:02.066 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-02-19T21:49:02.066 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:49:02.066 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:49:02.066 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:49:02.066 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:49:02.067 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T21:49:02.077 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T21:49:02.077 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi143.front.sepia.ceph.com 2024-02-19T21:49:02.077 DEBUG:teuthology.orchestra.run.smithi143:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T21:49:02.078 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi173.front.sepia.ceph.com 2024-02-19T21:49:02.078 DEBUG:teuthology.orchestra.run.smithi173:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T21:49:02.866 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Feb 2024 09:47:27 PM UTC. 2024-02-19T21:49:02.889 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Feb 2024 09:47:27 PM UTC. 2024-02-19T21:49:04.880 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:49:04.881 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:49:04.881 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-02-19T21:49:04.881 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:49:04.881 INFO:teuthology.orchestra.run.smithi143.stdout:Resetting modules: 2024-02-19T21:49:04.881 INFO:teuthology.orchestra.run.smithi143.stdout: container-tools 2024-02-19T21:49:04.882 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:49:04.882 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:49:04.882 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:49:04.882 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:49:04.903 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:49:04.904 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:49:04.904 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-19T21:49:04.904 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:49:04.904 INFO:teuthology.orchestra.run.smithi173.stdout:Resetting modules: 2024-02-19T21:49:04.904 INFO:teuthology.orchestra.run.smithi173.stdout: container-tools 2024-02-19T21:49:04.904 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:49:04.904 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:49:04.904 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:49:04.904 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:49:05.134 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:49:05.154 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:49:05.989 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Feb 2024 09:47:27 PM UTC. 2024-02-19T21:49:06.012 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Feb 2024 09:47:27 PM UTC. 2024-02-19T21:49:08.004 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:49:08.031 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:49:08.039 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:49:08.039 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-02-19T21:49:08.039 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout:Installing group/module packages: 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-02-19T21:49:08.040 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout:Installing weak dependencies: 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout:Installing module profiles: 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout: container-tools/common 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout:Enabling module streams: 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout: container-tools rhel8 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout:Install 13 Packages 2024-02-19T21:49:08.041 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:49:08.042 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 28 M 2024-02-19T21:49:08.042 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 92 M 2024-02-19T21:49:08.042 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:49:08.066 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:49:08.066 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-19T21:49:08.066 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:49:08.067 INFO:teuthology.orchestra.run.smithi173.stdout:Installing group/module packages: 2024-02-19T21:49:08.067 INFO:teuthology.orchestra.run.smithi173.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T21:49:08.067 INFO:teuthology.orchestra.run.smithi173.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T21:49:08.067 INFO:teuthology.orchestra.run.smithi173.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T21:49:08.067 INFO:teuthology.orchestra.run.smithi173.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T21:49:08.067 INFO:teuthology.orchestra.run.smithi173.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T21:49:08.067 INFO:teuthology.orchestra.run.smithi173.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T21:49:08.067 INFO:teuthology.orchestra.run.smithi173.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T21:49:08.067 INFO:teuthology.orchestra.run.smithi173.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout:Installing module profiles: 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout: container-tools/common 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout:Enabling module streams: 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout: container-tools rhel8 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:49:08.068 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:49:08.069 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-02-19T21:49:08.069 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:49:08.069 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 28 M 2024-02-19T21:49:08.069 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 92 M 2024-02-19T21:49:08.069 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:49:08.551 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 2.3 MB/s | 688 kB 00:00 2024-02-19T21:49:08.593 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 916 kB/s | 38 kB 00:00 2024-02-19T21:49:08.760 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.0 MB/s | 1.0 MB 00:00 2024-02-19T21:49:08.777 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 1.4 MB/s | 257 kB 00:00 2024-02-19T21:49:08.886 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.6 MB/s | 180 kB 00:00 2024-02-19T21:49:08.928 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 598 kB/s | 25 kB 00:00 2024-02-19T21:49:08.978 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.8 MB/s | 94 kB 00:00 2024-02-19T21:49:09.523 INFO:teuthology.orchestra.run.smithi143.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 2.4 MB/s | 688 kB 00:00 2024-02-19T21:49:09.556 INFO:teuthology.orchestra.run.smithi143.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.2 MB/s | 1.0 MB 00:00 2024-02-19T21:49:09.590 INFO:teuthology.orchestra.run.smithi143.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 569 kB/s | 38 kB 00:00 2024-02-19T21:49:09.623 INFO:teuthology.orchestra.run.smithi143.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 3.8 MB/s | 257 kB 00:00 2024-02-19T21:49:09.857 INFO:teuthology.orchestra.run.smithi143.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 773 kB/s | 180 kB 00:00 2024-02-19T21:49:09.907 INFO:teuthology.orchestra.run.smithi143.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 496 kB/s | 25 kB 00:00 2024-02-19T21:49:09.949 INFO:teuthology.orchestra.run.smithi143.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 2.2 MB/s | 94 kB 00:00 2024-02-19T21:49:11.624 INFO:teuthology.orchestra.run.smithi143.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 2.1 MB/s | 4.3 MB 00:02 2024-02-19T21:49:11.720 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 1.5 MB/s | 4.3 MB 00:02 2024-02-19T21:49:12.891 INFO:teuthology.orchestra.run.smithi143.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 2.6 MB/s | 9.7 MB 00:03 2024-02-19T21:49:12.949 INFO:teuthology.orchestra.run.smithi143.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 908 kB/s | 53 kB 00:00 2024-02-19T21:49:13.041 INFO:teuthology.orchestra.run.smithi143.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 447 kB/s | 41 kB 00:00 2024-02-19T21:49:13.320 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): toolbox-0.0.99.5-1.module_el8+847+7863d 1.6 MB/s | 2.5 MB 00:01 2024-02-19T21:49:13.396 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 702 kB/s | 53 kB 00:00 2024-02-19T21:49:13.466 INFO:teuthology.orchestra.run.smithi143.stdout:(12/13): skopeo-1.14.3-0.1.module_el8+885+7da14 2.5 MB/s | 8.8 MB 00:03 2024-02-19T21:49:13.521 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): buildah-1.33.5-1.module_el8+885+7da147 1.8 MB/s | 9.7 MB 00:05 2024-02-19T21:49:13.546 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 273 kB/s | 41 kB 00:00 2024-02-19T21:49:13.626 INFO:teuthology.orchestra.run.smithi143.stdout:(13/13): toolbox-0.0.99.5-1.module_el8+847+7863 1.3 MB/s | 2.5 MB 00:02 2024-02-19T21:49:13.630 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:49:13.630 INFO:teuthology.orchestra.run.smithi143.stdout:Total 4.9 MB/s | 28 MB 00:05 2024-02-19T21:49:13.906 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:49:13.926 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:49:13.926 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:49:14.163 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 1.7 MB/s | 8.8 MB 00:05 2024-02-19T21:49:14.163 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:49:14.163 INFO:teuthology.orchestra.run.smithi173.stdout:Total 4.5 MB/s | 28 MB 00:06 2024-02-19T21:49:14.206 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:49:14.209 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:49:14.438 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:49:14.459 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:49:14.459 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:49:14.722 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:49:14.725 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:49:14.915 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:49:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:49:15.894 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T21:49:16.092 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T21:49:16.245 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T21:49:16.428 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T21:49:16.505 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T21:49:16.680 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T21:49:16.725 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T21:49:16.903 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T21:49:17.122 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T21:49:17.249 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T21:49:17.372 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T21:49:17.999 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T21:49:18.199 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T21:49:18.415 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T21:49:18.618 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T21:49:19.012 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T21:49:19.135 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T21:49:19.286 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T21:49:19.439 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T21:49:19.497 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T21:49:19.649 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T21:49:19.835 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T21:49:19.978 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T21:49:20.261 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T21:49:20.261 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T21:49:20.261 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T21:49:20.261 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T21:49:20.261 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T21:49:20.261 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T21:49:20.261 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T21:49:20.261 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T21:49:20.262 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T21:49:20.262 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T21:49:20.262 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T21:49:20.262 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T21:49:20.262 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T21:49:20.552 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T21:49:20.781 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T21:49:20.949 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T21:49:20.950 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:49:20.950 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:49:20.950 INFO:teuthology.orchestra.run.smithi143.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T21:49:20.950 INFO:teuthology.orchestra.run.smithi143.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T21:49:20.950 INFO:teuthology.orchestra.run.smithi143.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T21:49:20.950 INFO:teuthology.orchestra.run.smithi143.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T21:49:20.950 INFO:teuthology.orchestra.run.smithi143.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T21:49:20.951 INFO:teuthology.orchestra.run.smithi143.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T21:49:20.951 INFO:teuthology.orchestra.run.smithi143.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T21:49:20.951 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T21:49:20.951 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T21:49:20.951 INFO:teuthology.orchestra.run.smithi143.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T21:49:20.951 INFO:teuthology.orchestra.run.smithi143.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T21:49:20.951 INFO:teuthology.orchestra.run.smithi143.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T21:49:20.951 INFO:teuthology.orchestra.run.smithi143.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T21:49:20.951 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:49:20.951 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:49:20.977 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T21:49:21.213 DEBUG:teuthology.parallel:result is None 2024-02-19T21:49:21.395 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T21:49:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T21:49:21.925 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T21:49:21.926 INFO:teuthology.orchestra.run.smithi173.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T21:49:21.927 INFO:teuthology.orchestra.run.smithi173.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T21:49:21.927 INFO:teuthology.orchestra.run.smithi173.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T21:49:21.927 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:49:21.927 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:49:22.224 DEBUG:teuthology.parallel:result is None 2024-02-19T21:49:22.224 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T21:49:22.234 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T21:49:22.234 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi143.front.sepia.ceph.com 2024-02-19T21:49:22.235 DEBUG:teuthology.orchestra.run.smithi143:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T21:49:22.235 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi173.front.sepia.ceph.com 2024-02-19T21:49:22.235 DEBUG:teuthology.orchestra.run.smithi173:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T21:49:22.395 DEBUG:teuthology.parallel:result is None 2024-02-19T21:49:22.444 DEBUG:teuthology.parallel:result is None 2024-02-19T21:49:22.444 INFO:teuthology.run_tasks:Running task install... 2024-02-19T21:49:22.454 DEBUG:teuthology.task.install:project ceph 2024-02-19T21:49:22.454 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'}} 2024-02-19T21:49:22.454 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T21:49:22.454 INFO:teuthology.task.install:Using flavor: default 2024-02-19T21:49:22.463 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T21:49:22.463 INFO:teuthology.task.install:extra packages: [] 2024-02-19T21:49:22.463 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T21:49:22.463 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:49:22.463 INFO:teuthology.packaging:ref: None 2024-02-19T21:49:22.464 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:49:22.464 INFO:teuthology.packaging:branch: None 2024-02-19T21:49:22.464 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:49:22.656 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T21:49:22.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T21:49:22.660 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T21:49:22.660 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:49:22.660 INFO:teuthology.packaging:ref: None 2024-02-19T21:49:22.660 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:49:22.660 INFO:teuthology.packaging:branch: None 2024-02-19T21:49:22.660 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:49:22.661 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T21:49:22.881 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T21:49:22.881 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T21:49:22.894 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T21:49:22.894 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T21:49:22.924 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T21:49:22.924 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-02-19T21:49:22.924 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T21:49:22.931 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T21:49:22.931 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-19T21:49:22.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T21:49:22.958 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T21:49:22.958 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:49:22.958 INFO:teuthology.packaging:ref: None 2024-02-19T21:49:22.958 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:49:22.959 INFO:teuthology.packaging:branch: None 2024-02-19T21:49:22.959 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:49:22.959 DEBUG:teuthology.orchestra.run.smithi143:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T21:49:22.960 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T21:49:22.960 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:49:22.960 INFO:teuthology.packaging:ref: None 2024-02-19T21:49:22.960 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:49:22.960 INFO:teuthology.packaging:branch: None 2024-02-19T21:49:22.960 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:49:22.961 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T21:49:23.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T21:49:23.030 DEBUG:teuthology.orchestra.run.smithi143:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T21:49:23.105 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T21:49:23.112 DEBUG:teuthology.orchestra.run.smithi143:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T21:49:23.171 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-02-19T21:49:23.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-02-19T21:49:23.181 INFO:teuthology.orchestra.run.smithi143.stdout:check_obsoletes = 1 2024-02-19T21:49:23.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum clean all 2024-02-19T21:49:23.439 INFO:teuthology.orchestra.run.smithi173.stdout:76 files removed 2024-02-19T21:49:23.453 INFO:teuthology.orchestra.run.smithi143.stdout:76 files removed 2024-02-19T21:49:23.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-02-19T21:49:23.481 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-radosgw 2024-02-19T21:49:23.948 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 983 kB/s | 74 kB 00:00 2024-02-19T21:49:23.965 INFO:teuthology.orchestra.run.smithi143.stdout:ceph packages for x86_64 966 kB/s | 74 kB 00:00 2024-02-19T21:49:24.073 INFO:teuthology.orchestra.run.smithi143.stdout:ceph noarch packages 214 kB/s | 15 kB 00:00 2024-02-19T21:49:24.083 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 156 kB/s | 15 kB 00:00 2024-02-19T21:49:24.193 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 21 kB/s | 1.7 kB 00:00 2024-02-19T21:49:24.267 INFO:teuthology.orchestra.run.smithi143.stdout:ceph source packages 11 kB/s | 1.7 kB 00:00 2024-02-19T21:49:31.743 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - AppStream 3.6 MB/s | 27 MB 00:07 2024-02-19T21:49:34.066 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 8 - AppStream 2.8 MB/s | 27 MB 00:09 2024-02-19T21:49:37.419 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-02-19T21:49:39.203 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras 103 kB/s | 18 kB 00:00 2024-02-19T21:49:39.874 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 10 MB 00:00 2024-02-19T21:49:39.903 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2024-02-19T21:49:40.996 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - PowerTools 4.8 MB/s | 4.8 MB 00:01 2024-02-19T21:49:41.591 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 8 - Extras 133 kB/s | 18 kB 00:00 2024-02-19T21:49:42.156 INFO:teuthology.orchestra.run.smithi173.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-02-19T21:49:43.151 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 8 - Extras common packages 4.8 kB/s | 6.9 kB 00:01 2024-02-19T21:49:43.773 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-02-19T21:49:44.276 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 8 - PowerTools 4.7 MB/s | 4.8 MB 00:01 2024-02-19T21:49:45.494 INFO:teuthology.orchestra.run.smithi143.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-02-19T21:49:47.607 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 295 kB/s | 24 kB 00:00 2024-02-19T21:49:47.811 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 16 MB 00:02 2024-02-19T21:49:51.137 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:49:51.140 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:49:51.140 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-19T21:49:51.140 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T21:49:51.141 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T21:49:51.142 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T21:49:51.143 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T21:49:51.143 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T21:49:51.143 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T21:49:51.143 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T21:49:51.143 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T21:49:51.143 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:49:51.143 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:49:51.143 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:49:51.143 INFO:teuthology.orchestra.run.smithi173.stdout:Install 25 Packages 2024-02-19T21:49:51.143 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-02-19T21:49:51.143 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:49:51.144 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 59 M 2024-02-19T21:49:51.144 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:49:51.579 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 595 kB/s | 24 kB 00:00 2024-02-19T21:49:51.783 INFO:teuthology.orchestra.run.smithi173.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 46 MB/s | 6.0 MB 00:00 2024-02-19T21:49:51.800 INFO:teuthology.orchestra.run.smithi173.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2024-02-19T21:49:52.034 INFO:teuthology.orchestra.run.smithi173.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 801 kB 00:00 2024-02-19T21:49:52.247 INFO:teuthology.orchestra.run.smithi173.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 2.2 MB/s | 486 kB 00:00 2024-02-19T21:49:52.794 INFO:teuthology.orchestra.run.smithi173.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 7.5 MB/s | 4.1 MB 00:00 2024-02-19T21:49:52.852 INFO:teuthology.orchestra.run.smithi173.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 764 kB/s | 44 kB 00:00 2024-02-19T21:49:52.878 INFO:teuthology.orchestra.run.smithi173.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-19T21:49:52.903 INFO:teuthology.orchestra.run.smithi173.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-02-19T21:49:52.936 INFO:teuthology.orchestra.run.smithi173.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-19T21:49:52.970 INFO:teuthology.orchestra.run.smithi173.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-02-19T21:49:52.995 INFO:teuthology.orchestra.run.smithi173.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-19T21:49:53.180 INFO:teuthology.orchestra.run.smithi173.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 13 MB 00:01 2024-02-19T21:49:53.204 INFO:teuthology.orchestra.run.smithi173.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 696 kB/s | 145 kB 00:00 2024-02-19T21:49:53.472 INFO:teuthology.orchestra.run.smithi173.stdout:(14/27): ceph-common-16.2.4-0.el8.x86_64.rpm 13 MB/s | 24 MB 00:01 2024-02-19T21:49:53.547 INFO:teuthology.orchestra.run.smithi173.stdout:(15/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 759 kB/s | 259 kB 00:00 2024-02-19T21:49:53.572 INFO:teuthology.orchestra.run.smithi173.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 439 kB/s | 44 kB 00:00 2024-02-19T21:49:53.639 INFO:teuthology.orchestra.run.smithi173.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 460 kB/s | 42 kB 00:00 2024-02-19T21:49:53.664 INFO:teuthology.orchestra.run.smithi173.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 511 kB/s | 47 kB 00:00 2024-02-19T21:49:53.691 INFO:teuthology.orchestra.run.smithi173.stdout:(19/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.3 MB/s | 677 kB 00:00 2024-02-19T21:49:53.715 INFO:teuthology.orchestra.run.smithi173.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.0 MB/s | 80 kB 00:00 2024-02-19T21:49:53.940 INFO:teuthology.orchestra.run.smithi173.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 422 kB/s | 116 kB 00:00 2024-02-19T21:49:53.965 INFO:teuthology.orchestra.run.smithi173.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 723 kB/s | 181 kB 00:00 2024-02-19T21:49:53.991 INFO:teuthology.orchestra.run.smithi173.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2024-02-19T21:49:54.016 INFO:teuthology.orchestra.run.smithi173.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 780 kB/s | 59 kB 00:00 2024-02-19T21:49:54.041 INFO:teuthology.orchestra.run.smithi173.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 994 kB/s | 75 kB 00:00 2024-02-19T21:49:54.125 INFO:teuthology.orchestra.run.smithi173.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-02-19T21:49:54.219 INFO:teuthology.orchestra.run.smithi173.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 20 MB/s | 4.0 MB 00:00 2024-02-19T21:49:54.224 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:49:54.224 INFO:teuthology.orchestra.run.smithi173.stdout:Total 19 MB/s | 59 MB 00:03 2024-02-19T21:49:54.294 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:49:54.343 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:49:54.343 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:49:54.681 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:49:54.684 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:49:55.083 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:49:55.086 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:49:55.086 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-02-19T21:49:55.086 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:49:55.086 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrading: 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T21:49:55.087 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T21:49:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T21:49:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T21:49:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T21:49:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T21:49:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T21:49:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T21:49:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T21:49:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T21:49:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T21:49:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T21:49:55.088 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T21:49:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T21:49:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T21:49:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T21:49:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T21:49:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T21:49:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:49:55.089 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:49:55.089 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:49:55.089 INFO:teuthology.orchestra.run.smithi143.stdout:Install 25 Packages 2024-02-19T21:49:55.089 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrade 2 Packages 2024-02-19T21:49:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:49:55.090 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 59 M 2024-02-19T21:49:55.090 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:49:55.597 INFO:teuthology.orchestra.run.smithi143.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 40 MB/s | 6.0 MB 00:00 2024-02-19T21:49:55.622 INFO:teuthology.orchestra.run.smithi143.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-02-19T21:49:55.664 INFO:teuthology.orchestra.run.smithi143.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-02-19T21:49:55.698 INFO:teuthology.orchestra.run.smithi143.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-19T21:49:55.790 INFO:teuthology.orchestra.run.smithi143.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2024-02-19T21:49:55.815 INFO:teuthology.orchestra.run.smithi143.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-02-19T21:49:55.831 INFO:teuthology.orchestra.run.smithi143.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 4.7 MB/s | 80 kB 00:00 2024-02-19T21:49:55.854 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:49:55.965 INFO:teuthology.orchestra.run.smithi143.stdout:(8/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 24 MB/s | 13 MB 00:00 2024-02-19T21:49:55.991 INFO:teuthology.orchestra.run.smithi143.stdout:(9/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 1.3 MB/s | 213 kB 00:00 2024-02-19T21:49:56.016 INFO:teuthology.orchestra.run.smithi143.stdout:(10/27): python3-rados-16.2.4-0.el8.x86_64.rpm 7.5 MB/s | 387 kB 00:00 2024-02-19T21:49:56.041 INFO:teuthology.orchestra.run.smithi143.stdout:(11/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 7.2 MB/s | 368 kB 00:00 2024-02-19T21:49:56.090 INFO:teuthology.orchestra.run.smithi143.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 114 kB 00:00 2024-02-19T21:49:56.102 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T21:49:56.349 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T21:49:56.358 INFO:teuthology.orchestra.run.smithi143.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-02-19T21:49:56.442 INFO:teuthology.orchestra.run.smithi143.stdout:(14/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.9 MB/s | 677 kB 00:00 2024-02-19T21:49:56.484 INFO:teuthology.orchestra.run.smithi143.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2024-02-19T21:49:56.509 INFO:teuthology.orchestra.run.smithi143.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.7 MB/s | 259 kB 00:00 2024-02-19T21:49:56.522 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T21:49:56.568 INFO:teuthology.orchestra.run.smithi143.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 506 kB/s | 42 kB 00:00 2024-02-19T21:49:56.593 INFO:teuthology.orchestra.run.smithi143.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 564 kB/s | 47 kB 00:00 2024-02-19T21:49:56.618 INFO:teuthology.orchestra.run.smithi143.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.6 MB/s | 80 kB 00:00 2024-02-19T21:49:56.969 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T21:49:57.002 INFO:teuthology.orchestra.run.smithi143.stdout:(20/27): fmt-6.2.1-1.el8.x86_64.rpm 285 kB/s | 116 kB 00:00 2024-02-19T21:49:57.061 INFO:teuthology.orchestra.run.smithi143.stdout:(21/27): gperftools-libs-2.7-9.el8.x86_64.rpm 693 kB/s | 306 kB 00:00 2024-02-19T21:49:57.161 INFO:teuthology.orchestra.run.smithi143.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 588 kB/s | 59 kB 00:00 2024-02-19T21:49:57.163 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:49:57.195 INFO:teuthology.orchestra.run.smithi143.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 940 kB/s | 181 kB 00:00 2024-02-19T21:49:57.211 INFO:teuthology.orchestra.run.smithi143.stdout:(24/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 124 kB/s | 145 kB 00:01 2024-02-19T21:49:57.253 INFO:teuthology.orchestra.run.smithi143.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 812 kB/s | 75 kB 00:00 2024-02-19T21:49:57.267 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:49:57.320 INFO:teuthology.orchestra.run.smithi143.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2024-02-19T21:49:57.379 INFO:teuthology.orchestra.run.smithi143.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2024-02-19T21:49:57.381 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:49:57.381 INFO:teuthology.orchestra.run.smithi143.stdout:Total 26 MB/s | 59 MB 00:02 2024-02-19T21:49:57.453 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:49:57.502 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:49:57.502 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:49:57.553 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T21:49:57.739 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:49:57.804 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:49:57.846 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:49:57.849 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:49:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T21:49:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T21:49:58.830 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:49:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:49:58.976 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:49:59.144 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T21:49:59.181 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T21:49:59.313 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T21:49:59.394 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T21:49:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T21:49:59.554 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T21:49:59.560 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T21:49:59.773 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T21:49:59.930 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T21:49:59.979 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T21:50:00.004 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T21:50:00.171 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:50:00.180 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:50:00.253 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:50:00.512 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T21:50:00.597 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:50:00.671 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:50:00.732 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:50:00.830 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T21:50:00.932 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T21:50:00.955 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T21:50:01.168 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T21:50:01.390 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T21:50:01.533 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T21:50:01.588 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T21:50:01.670 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T21:50:01.687 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:50:01.781 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:50:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T21:50:01.984 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T21:50:02.154 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T21:50:02.178 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T21:50:02.204 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T21:50:02.362 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T21:50:02.384 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T21:50:02.588 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T21:50:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T21:50:02.754 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T21:50:02.814 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T21:50:02.970 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:50:03.409 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:50:03.612 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T21:50:03.714 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T21:50:03.932 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T21:50:04.172 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T21:50:04.353 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T21:50:04.433 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T21:50:04.647 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T21:50:04.880 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T21:50:05.053 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T21:50:05.146 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:50:05.279 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T21:50:05.413 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:50:06.111 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:50:06.442 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T21:50:06.551 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T21:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:50:07.886 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:50:08.120 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:50:08.826 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:50:09.119 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T21:50:09.216 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T21:50:09.361 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:50:26.157 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:50:26.158 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-02-19T21:50:26.158 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-02-19T21:50:26.158 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /dev 2024-02-19T21:50:26.158 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /run 2024-02-19T21:50:26.158 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-02-19T21:50:26.158 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-02-19T21:50:26.158 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-02-19T21:50:26.158 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-02-19T21:50:26.158 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /dev 2024-02-19T21:50:26.158 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:50:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T21:50:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T21:50:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T21:50:27.853 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T21:50:27.853 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:50:28.576 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:50:28.577 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /sys 2024-02-19T21:50:28.577 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /proc 2024-02-19T21:50:28.577 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /dev 2024-02-19T21:50:28.577 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /run 2024-02-19T21:50:28.577 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /mnt 2024-02-19T21:50:28.577 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /var/tmp 2024-02-19T21:50:28.577 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /home 2024-02-19T21:50:28.577 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /tmp 2024-02-19T21:50:28.577 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /dev 2024-02-19T21:50:28.577 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:50:28.591 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:50:28.591 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T21:50:28.591 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T21:50:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T21:50:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T21:50:29.439 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:50:29.440 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:50:29.440 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-02-19T21:50:29.440 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.440 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:50:29.440 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.440 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.440 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.440 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.440 INFO:teuthology.orchestra.run.smithi173.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T21:50:29.440 INFO:teuthology.orchestra.run.smithi173.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.441 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.442 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.442 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T21:50:29.442 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.442 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.442 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:29.442 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:50:29.442 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:50:29.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-02-19T21:50:29.932 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T21:50:30.000 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T21:50:30.000 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T21:50:30.202 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T21:50:30.202 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:50:30.523 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:43 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:50:30.909 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:50:30.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T21:50:30.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T21:50:30.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T21:50:30.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:50:30.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T21:50:30.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T21:50:30.911 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T21:50:30.911 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T21:50:30.911 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T21:50:30.911 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T21:50:30.911 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T21:50:30.911 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T21:50:30.911 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T21:50:30.911 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:50:30.911 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T21:50:30.911 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T21:50:31.828 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:50:31.828 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:50:31.828 INFO:teuthology.orchestra.run.smithi143.stdout:Upgraded: 2024-02-19T21:50:31.828 INFO:teuthology.orchestra.run.smithi143.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.828 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:50:31.828 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.828 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.828 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.828 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T21:50:31.829 INFO:teuthology.orchestra.run.smithi143.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T21:50:31.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T21:50:31.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:31.830 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:50:31.830 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:50:32.074 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:50:32.075 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:50:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-19T21:50:32.075 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:50:32.075 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:50:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T21:50:32.075 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-19T21:50:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T21:50:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T21:50:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T21:50:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T21:50:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T21:50:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:50:32.076 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:50:32.076 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:50:32.076 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-02-19T21:50:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:50:32.076 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 53 M 2024-02-19T21:50:32.077 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 229 M 2024-02-19T21:50:32.077 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:50:32.120 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-test 2024-02-19T21:50:32.318 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 169 kB 00:00 2024-02-19T21:50:32.419 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.3 MB/s | 203 kB 00:00 2024-02-19T21:50:32.444 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-02-19T21:50:32.494 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-02-19T21:50:32.611 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.6 MB/s | 323 kB 00:00 2024-02-19T21:50:32.879 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:41 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:50:33.128 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 60 MB/s | 52 MB 00:00 2024-02-19T21:50:33.130 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:50:33.131 INFO:teuthology.orchestra.run.smithi173.stdout:Total 50 MB/s | 53 MB 00:01 2024-02-19T21:50:33.173 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:50:33.194 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:50:33.194 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:50:33.515 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:50:33.518 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:50:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:50:34.244 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T21:50:34.419 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T21:50:34.432 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:50:34.433 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:50:34.433 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-02-19T21:50:34.433 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:50:34.433 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:50:34.433 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T21:50:34.433 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-02-19T21:50:34.433 INFO:teuthology.orchestra.run.smithi143.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T21:50:34.433 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T21:50:34.433 INFO:teuthology.orchestra.run.smithi143.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T21:50:34.433 INFO:teuthology.orchestra.run.smithi143.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T21:50:34.433 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T21:50:34.434 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:50:34.434 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:50:34.434 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:50:34.434 INFO:teuthology.orchestra.run.smithi143.stdout:Install 6 Packages 2024-02-19T21:50:34.434 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:50:34.434 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 53 M 2024-02-19T21:50:34.434 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 229 M 2024-02-19T21:50:34.435 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:50:34.600 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T21:50:34.682 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T21:50:34.708 INFO:teuthology.orchestra.run.smithi143.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 169 kB 00:00 2024-02-19T21:50:34.742 INFO:teuthology.orchestra.run.smithi143.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.8 MB/s | 203 kB 00:00 2024-02-19T21:50:34.817 INFO:teuthology.orchestra.run.smithi143.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2024-02-19T21:50:34.859 INFO:teuthology.orchestra.run.smithi143.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-02-19T21:50:34.884 INFO:teuthology.orchestra.run.smithi143.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-02-19T21:50:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T21:50:35.033 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T21:50:35.468 INFO:teuthology.orchestra.run.smithi143.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 62 MB/s | 52 MB 00:00 2024-02-19T21:50:35.470 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:50:35.471 INFO:teuthology.orchestra.run.smithi143.stdout:Total 51 MB/s | 53 MB 00:01 2024-02-19T21:50:35.514 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:50:35.535 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:50:35.535 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:50:35.844 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:50:35.847 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:50:36.335 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:50:36.553 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T21:50:36.757 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T21:50:36.957 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T21:50:37.040 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T21:50:37.223 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T21:50:37.365 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T21:50:39.844 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T21:50:40.098 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T21:50:40.392 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T21:50:40.392 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T21:50:40.392 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T21:50:40.392 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T21:50:40.393 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T21:50:40.393 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T21:50:40.921 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T21:50:40.921 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:50:40.921 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:50:40.921 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T21:50:40.921 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T21:50:40.921 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T21:50:40.921 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:50:40.921 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:50:41.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-02-19T21:50:41.960 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:54 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:50:42.207 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T21:50:42.446 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T21:50:42.760 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T21:50:42.761 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T21:50:42.761 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T21:50:42.761 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T21:50:42.761 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T21:50:42.761 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T21:50:43.202 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T21:50:43.203 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:50:43.203 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:50:43.203 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T21:50:43.203 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T21:50:43.203 INFO:teuthology.orchestra.run.smithi143.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T21:50:43.203 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:50:43.203 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:50:43.439 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph 2024-02-19T21:50:43.518 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:50:43.521 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:50:43.521 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-19T21:50:43.521 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:50:43.521 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:50:43.521 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T21:50:43.521 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-19T21:50:43.521 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T21:50:43.521 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T21:50:43.521 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T21:50:43.521 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T21:50:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:50:43.523 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:50:43.524 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:50:43.524 INFO:teuthology.orchestra.run.smithi173.stdout:Install 32 Packages 2024-02-19T21:50:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:50:43.524 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 31 M 2024-02-19T21:50:43.524 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 111 M 2024-02-19T21:50:43.524 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:50:43.847 INFO:teuthology.orchestra.run.smithi173.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 123 kB/s | 6.3 kB 00:00 2024-02-19T21:50:43.897 INFO:teuthology.orchestra.run.smithi173.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-02-19T21:50:43.964 INFO:teuthology.orchestra.run.smithi173.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-02-19T21:50:43.989 INFO:teuthology.orchestra.run.smithi173.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-02-19T21:50:44.074 INFO:teuthology.orchestra.run.smithi173.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-02-19T21:50:44.099 INFO:teuthology.orchestra.run.smithi173.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2024-02-19T21:50:44.166 INFO:teuthology.orchestra.run.smithi173.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2024-02-19T21:50:44.191 INFO:teuthology.orchestra.run.smithi173.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 331 kB/s | 39 kB 00:00 2024-02-19T21:50:44.205 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:53 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:50:44.216 INFO:teuthology.orchestra.run.smithi173.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-02-19T21:50:44.417 INFO:teuthology.orchestra.run.smithi173.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 35 MB/s | 18 MB 00:00 2024-02-19T21:50:44.500 INFO:teuthology.orchestra.run.smithi173.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 877 kB/s | 249 kB 00:00 2024-02-19T21:50:44.525 INFO:teuthology.orchestra.run.smithi173.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2024-02-19T21:50:44.559 INFO:teuthology.orchestra.run.smithi173.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 4.4 MB/s | 641 kB 00:00 2024-02-19T21:50:44.601 INFO:teuthology.orchestra.run.smithi173.stdout:(14/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.1 MB/s | 457 kB 00:00 2024-02-19T21:50:44.701 INFO:teuthology.orchestra.run.smithi173.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 248 kB/s | 44 kB 00:00 2024-02-19T21:50:44.760 INFO:teuthology.orchestra.run.smithi173.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 944 kB/s | 185 kB 00:00 2024-02-19T21:50:44.815 INFO:teuthology.orchestra.run.smithi173.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 828 kB/s | 173 kB 00:00 2024-02-19T21:50:44.844 INFO:teuthology.orchestra.run.smithi173.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2024-02-19T21:50:44.861 INFO:teuthology.orchestra.run.smithi173.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 386 kB/s | 18 kB 00:00 2024-02-19T21:50:44.877 INFO:teuthology.orchestra.run.smithi173.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.4 MB/s | 49 kB 00:00 2024-02-19T21:50:44.894 INFO:teuthology.orchestra.run.smithi173.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.7 MB/s | 59 kB 00:00 2024-02-19T21:50:44.920 INFO:teuthology.orchestra.run.smithi173.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2024-02-19T21:50:44.937 INFO:teuthology.orchestra.run.smithi173.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 371 kB/s | 16 kB 00:00 2024-02-19T21:50:44.954 INFO:teuthology.orchestra.run.smithi173.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 554 kB/s | 19 kB 00:00 2024-02-19T21:50:44.980 INFO:teuthology.orchestra.run.smithi173.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2024-02-19T21:50:44.996 INFO:teuthology.orchestra.run.smithi173.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 403 kB/s | 24 kB 00:00 2024-02-19T21:50:45.013 INFO:teuthology.orchestra.run.smithi173.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 499 kB/s | 29 kB 00:00 2024-02-19T21:50:45.039 INFO:teuthology.orchestra.run.smithi173.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 469 kB/s | 27 kB 00:00 2024-02-19T21:50:45.081 INFO:teuthology.orchestra.run.smithi173.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.0 MB/s | 85 kB 00:00 2024-02-19T21:50:45.106 INFO:teuthology.orchestra.run.smithi173.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2024-02-19T21:50:45.152 INFO:teuthology.orchestra.run.smithi173.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2024-02-19T21:50:45.181 INFO:teuthology.orchestra.run.smithi173.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2024-02-19T21:50:45.185 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:50:45.185 INFO:teuthology.orchestra.run.smithi173.stdout:Total 19 MB/s | 31 MB 00:01 2024-02-19T21:50:45.265 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:50:45.304 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:50:45.304 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:50:45.632 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:50:45.635 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:50:45.768 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T21:50:45.772 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T21:50:45.773 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:50:45.774 INFO:teuthology.orchestra.run.smithi143.stdout:Install 32 Packages 2024-02-19T21:50:45.775 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:50:45.775 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 31 M 2024-02-19T21:50:45.775 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 111 M 2024-02-19T21:50:45.775 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:50:46.188 INFO:teuthology.orchestra.run.smithi143.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 124 kB/s | 6.3 kB 00:00 2024-02-19T21:50:46.238 INFO:teuthology.orchestra.run.smithi143.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-02-19T21:50:46.289 INFO:teuthology.orchestra.run.smithi143.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-02-19T21:50:46.314 INFO:teuthology.orchestra.run.smithi143.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-02-19T21:50:46.392 INFO:teuthology.orchestra.run.smithi143.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 25 MB/s | 5.0 MB 00:00 2024-02-19T21:50:46.448 INFO:teuthology.orchestra.run.smithi143.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 665 kB/s | 39 kB 00:00 2024-02-19T21:50:46.473 INFO:teuthology.orchestra.run.smithi143.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 989 kB/s | 157 kB 00:00 2024-02-19T21:50:46.674 INFO:teuthology.orchestra.run.smithi143.stdout:(8/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 42 MB/s | 18 MB 00:00 2024-02-19T21:50:46.724 INFO:teuthology.orchestra.run.smithi143.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 376 kB/s | 103 kB 00:00 2024-02-19T21:50:46.741 INFO:teuthology.orchestra.run.smithi143.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 213 kB/s | 57 kB 00:00 2024-02-19T21:50:46.794 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:50:46.825 INFO:teuthology.orchestra.run.smithi143.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.4 MB/s | 249 kB 00:00 2024-02-19T21:50:46.858 INFO:teuthology.orchestra.run.smithi143.stdout:(12/32): python3-cryptography-3.2.1-7.el8.x86_6 5.4 MB/s | 641 kB 00:00 2024-02-19T21:50:46.884 INFO:teuthology.orchestra.run.smithi143.stdout:(13/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.1 MB/s | 457 kB 00:00 2024-02-19T21:50:46.909 INFO:teuthology.orchestra.run.smithi143.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 1.3 MB/s | 109 kB 00:00 2024-02-19T21:50:47.045 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T21:50:47.268 INFO:teuthology.orchestra.run.smithi143.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 107 kB/s | 44 kB 00:00 2024-02-19T21:50:47.272 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T21:50:47.548 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T21:50:47.751 INFO:teuthology.orchestra.run.smithi143.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 205 kB/s | 173 kB 00:00 2024-02-19T21:50:47.779 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T21:50:47.827 INFO:teuthology.orchestra.run.smithi143.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 196 kB/s | 185 kB 00:00 2024-02-19T21:50:47.877 INFO:teuthology.orchestra.run.smithi143.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2024-02-19T21:50:47.952 INFO:teuthology.orchestra.run.smithi143.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 140 kB/s | 18 kB 00:00 2024-02-19T21:50:48.002 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T21:50:48.136 INFO:teuthology.orchestra.run.smithi143.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 188 kB/s | 49 kB 00:00 2024-02-19T21:50:48.183 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T21:50:48.286 INFO:teuthology.orchestra.run.smithi143.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 176 kB/s | 59 kB 00:00 2024-02-19T21:50:48.376 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T21:50:48.403 INFO:teuthology.orchestra.run.smithi143.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 134 kB/s | 16 kB 00:00 2024-02-19T21:50:48.520 INFO:teuthology.orchestra.run.smithi143.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 160 kB/s | 19 kB 00:00 2024-02-19T21:50:48.553 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T21:50:48.637 INFO:teuthology.orchestra.run.smithi143.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 205 kB/s | 24 kB 00:00 2024-02-19T21:50:48.696 INFO:teuthology.orchestra.run.smithi143.stdout:(25/32): python3-cherrypy-18.4.0-1.el8.noarch.r 269 kB/s | 384 kB 00:01 2024-02-19T21:50:48.754 INFO:teuthology.orchestra.run.smithi143.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 250 kB/s | 29 kB 00:00 2024-02-19T21:50:48.771 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T21:50:48.838 INFO:teuthology.orchestra.run.smithi143.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 194 kB/s | 27 kB 00:00 2024-02-19T21:50:49.017 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T21:50:49.063 INFO:teuthology.orchestra.run.smithi143.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 305 kB/s | 283 kB 00:00 2024-02-19T21:50:49.213 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T21:50:49.363 INFO:teuthology.orchestra.run.smithi143.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 285 kB/s | 85 kB 00:00 2024-02-19T21:50:49.486 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T21:50:49.488 INFO:teuthology.orchestra.run.smithi143.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2024-02-19T21:50:49.614 INFO:teuthology.orchestra.run.smithi143.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 324 kB/s | 251 kB 00:00 2024-02-19T21:50:49.639 INFO:teuthology.orchestra.run.smithi143.stdout:(32/32): python3-waitress-1.4.3-1.el8.noarch.rp 289 kB/s | 255 kB 00:00 2024-02-19T21:50:49.640 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:50:49.641 INFO:teuthology.orchestra.run.smithi143.stdout:Total 8.0 MB/s | 31 MB 00:03 2024-02-19T21:50:49.722 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:50:49.761 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:50:49.761 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:50:49.885 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T21:50:50.085 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:50:50.088 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:50:50.177 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T21:50:50.347 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T21:50:50.613 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T21:50:50.823 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T21:50:51.105 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T21:50:51.235 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:50:51.328 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T21:50:51.426 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T21:50:51.557 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T21:50:51.598 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T21:50:51.728 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T21:50:51.887 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T21:50:51.978 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T21:50:52.094 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T21:50:52.217 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T21:50:52.258 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T21:50:52.458 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T21:50:52.463 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T21:50:52.623 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T21:50:52.799 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T21:50:52.800 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T21:50:53.007 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T21:50:53.086 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T21:50:53.263 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T21:50:53.467 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T21:50:53.479 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T21:50:53.690 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T21:50:53.692 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T21:50:54.082 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T21:50:54.331 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T21:50:54.536 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T21:50:54.789 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T21:50:54.971 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T21:50:55.222 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T21:50:55.313 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T21:50:55.441 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T21:50:55.517 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T21:50:55.681 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T21:50:55.875 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T21:50:55.875 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T21:50:56.100 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T21:50:56.143 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T21:50:56.327 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T21:50:56.382 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T21:50:56.626 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T21:50:56.960 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T21:50:57.234 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T21:50:57.264 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T21:50:57.285 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T21:50:57.447 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T21:50:57.650 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T21:50:57.827 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T21:50:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T21:50:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T21:50:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T21:50:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T21:50:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T21:50:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T21:50:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T21:50:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T21:50:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T21:50:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T21:50:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T21:50:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T21:50:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T21:50:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T21:50:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T21:50:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T21:50:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T21:50:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T21:50:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T21:50:58.873 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T21:50:58.873 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:50:58.873 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:50:58.873 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:58.873 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:58.873 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:58.873 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T21:50:58.874 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T21:50:58.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T21:50:58.876 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:50:58.876 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:50:59.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-02-19T21:50:59.376 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T21:50:59.627 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T21:50:59.938 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:12 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:51:00.062 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T21:51:00.327 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T21:51:00.550 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T21:51:00.745 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T21:51:00.763 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T21:51:00.936 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T21:51:01.345 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T21:51:01.345 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T21:51:01.345 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T21:51:01.345 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T21:51:01.345 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T21:51:01.345 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T21:51:01.345 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T21:51:01.345 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T21:51:01.345 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T21:51:01.345 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T21:51:01.345 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T21:51:01.346 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T21:51:01.347 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T21:51:01.347 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T21:51:01.347 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T21:51:01.347 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T21:51:01.347 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T21:51:01.347 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T21:51:01.347 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T21:51:01.440 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:51:01.487 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:51:01.489 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-19T21:51:01.489 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:51:01.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-02-19T21:51:02.234 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T21:51:02.234 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:02.234 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:51:02.234 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T21:51:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T21:51:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T21:51:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T21:51:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T21:51:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T21:51:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T21:51:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T21:51:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T21:51:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T21:51:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T21:51:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T21:51:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T21:51:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T21:51:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T21:51:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T21:51:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T21:51:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T21:51:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T21:51:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:02.237 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:51:02.370 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:15 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:51:02.513 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-base 2024-02-19T21:51:03.269 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:12 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:51:03.919 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:51:03.919 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:03.920 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-19T21:51:03.920 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:03.920 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:51:03.920 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T21:51:03.920 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:03.920 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:51:03.920 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:03.920 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-19T21:51:03.921 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:03.921 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 71 k 2024-02-19T21:51:03.921 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 290 k 2024-02-19T21:51:03.921 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:51:03.962 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 71 kB 00:00 2024-02-19T21:51:03.963 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:03.963 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.6 MB/s | 71 kB 00:00 2024-02-19T21:51:03.963 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:51:03.970 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:51:03.970 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:51:03.972 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:51:03.975 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:51:04.018 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:51:04.523 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:51:04.711 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:51:04.781 INFO:teuthology.orchestra.run.smithi143.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:51:04.827 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:51:04.829 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-02-19T21:51:04.829 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:51:04.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install cephadm 2024-02-19T21:51:04.945 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:51:05.368 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:51:05.368 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:05.368 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:51:05.368 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T21:51:05.368 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:05.368 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:51:05.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T21:51:05.688 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:14 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:51:06.342 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:19 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:51:07.216 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:51:07.217 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:07.217 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-02-19T21:51:07.217 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:07.217 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:51:07.217 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T21:51:07.217 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:07.217 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:51:07.217 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:07.217 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-02-19T21:51:07.217 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:07.217 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 71 k 2024-02-19T21:51:07.218 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 290 k 2024-02-19T21:51:07.218 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:51:07.289 INFO:teuthology.orchestra.run.smithi143.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.0 MB/s | 71 kB 00:00 2024-02-19T21:51:07.290 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:07.290 INFO:teuthology.orchestra.run.smithi143.stdout:Total 985 kB/s | 71 kB 00:00 2024-02-19T21:51:07.290 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:51:07.296 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:51:07.297 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:51:07.298 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:51:07.301 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:51:07.349 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:51:07.776 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:51:07.890 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:51:07.891 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:07.891 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-02-19T21:51:07.891 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:07.891 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:51:07.891 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T21:51:07.891 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:07.891 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:51:07.891 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:07.891 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-19T21:51:07.891 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:07.892 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 163 k 2024-02-19T21:51:07.892 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 587 k 2024-02-19T21:51:07.892 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:51:07.980 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:51:08.208 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:51:08.512 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 263 kB/s | 163 kB 00:00 2024-02-19T21:51:08.512 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:08.513 INFO:teuthology.orchestra.run.smithi173.stdout:Total 262 kB/s | 163 kB 00:00 2024-02-19T21:51:08.513 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:51:08.526 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:51:08.526 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:51:08.578 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:51:08.581 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:51:08.597 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:51:08.597 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:08.597 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:51:08.597 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T21:51:08.597 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:08.597 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:51:09.391 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:51:09.786 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:51:10.174 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T21:51:10.226 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:51:10.608 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:51:10.608 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:10.608 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:51:10.608 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T21:51:10.608 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:10.608 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:51:10.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-02-19T21:51:10.939 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:19 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:51:11.598 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:24 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:51:12.496 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:51:12.497 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:12.497 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-02-19T21:51:12.497 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:12.497 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:51:12.497 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T21:51:12.497 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:12.497 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:51:12.497 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:12.498 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-02-19T21:51:12.498 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:12.498 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 163 k 2024-02-19T21:51:12.498 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 587 k 2024-02-19T21:51:12.498 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:51:12.545 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.4 MB/s | 163 kB 00:00 2024-02-19T21:51:12.545 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:12.546 INFO:teuthology.orchestra.run.smithi143.stdout:Total 3.3 MB/s | 163 kB 00:00 2024-02-19T21:51:12.546 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:51:12.556 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:51:12.556 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:51:12.604 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:51:12.607 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:51:12.721 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:51:12.876 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:51:13.096 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:51:13.142 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:51:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-19T21:51:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:51:13.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T21:51:13.291 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:51:13.656 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:51:13.656 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:13.656 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:51:13.656 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T21:51:13.656 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:13.656 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:51:13.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr 2024-02-19T21:51:14.002 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:27 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:51:14.660 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:23 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:51:15.556 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T21:51:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:15.558 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:51:15.558 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:15.558 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-02-19T21:51:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:15.558 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.0 M 2024-02-19T21:51:15.558 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 15 M 2024-02-19T21:51:15.559 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:51:15.829 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 469 kB/s | 24 kB 00:00 2024-02-19T21:51:15.854 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 117 kB/s | 8.9 kB 00:00 2024-02-19T21:51:15.938 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 17 MB/s | 2.7 MB 00:00 2024-02-19T21:51:15.988 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 268 kB/s | 43 kB 00:00 2024-02-19T21:51:16.178 INFO:teuthology.orchestra.run.smithi143.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:51:16.227 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:51:16.228 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-02-19T21:51:16.228 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:51:16.272 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 81 kB/s | 34 kB 00:00 2024-02-19T21:51:16.326 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T21:51:16.530 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 334 kB/s | 196 kB 00:00 2024-02-19T21:51:16.534 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:16.534 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.0 MB/s | 3.0 MB 00:00 2024-02-19T21:51:16.545 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:51:16.556 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:51:16.556 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:51:16.729 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:51:16.733 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:51:16.997 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:51:17.076 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:26 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:51:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T21:51:17.486 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T21:51:17.704 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T21:51:17.808 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T21:51:18.631 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:51:18.632 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:18.632 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-02-19T21:51:18.632 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:18.632 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:51:18.632 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T21:51:18.632 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-02-19T21:51:18.632 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T21:51:18.632 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T21:51:18.633 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T21:51:18.633 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T21:51:18.633 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T21:51:18.633 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:18.633 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:51:18.633 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:18.633 INFO:teuthology.orchestra.run.smithi143.stdout:Install 6 Packages 2024-02-19T21:51:18.633 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:18.634 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 3.0 M 2024-02-19T21:51:18.634 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 15 M 2024-02-19T21:51:18.634 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:51:18.801 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T21:51:19.032 INFO:teuthology.orchestra.run.smithi143.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 545 kB/s | 24 kB 00:00 2024-02-19T21:51:19.049 INFO:teuthology.orchestra.run.smithi143.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 146 kB/s | 8.9 kB 00:00 2024-02-19T21:51:19.096 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:51:19.116 INFO:teuthology.orchestra.run.smithi143.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 21 MB/s | 2.7 MB 00:00 2024-02-19T21:51:19.175 INFO:teuthology.orchestra.run.smithi143.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 299 kB/s | 43 kB 00:00 2024-02-19T21:51:19.317 INFO:teuthology.orchestra.run.smithi143.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 126 kB/s | 34 kB 00:00 2024-02-19T21:51:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:51:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T21:51:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T21:51:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T21:51:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T21:51:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T21:51:19.500 INFO:teuthology.orchestra.run.smithi143.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 518 kB/s | 196 kB 00:00 2024-02-19T21:51:19.504 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:19.504 INFO:teuthology.orchestra.run.smithi143.stdout:Total 3.4 MB/s | 3.0 MB 00:00 2024-02-19T21:51:19.514 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:51:19.523 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:51:19.524 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:51:19.699 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:51:19.702 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:51:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T21:51:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:19.811 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:51:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T21:51:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T21:51:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T21:51:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T21:51:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T21:51:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T21:51:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:19.812 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:51:19.934 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T21:51:19.972 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:51:20.206 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T21:51:20.425 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T21:51:20.627 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T21:51:20.688 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:33 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:51:20.764 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T21:51:21.772 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T21:51:22.031 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:51:22.246 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:51:22.247 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-02-19T21:51:22.247 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-19T21:51:22.247 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-02-19T21:51:22.248 INFO:teuthology.orchestra.run.smithi173.stdout:Install 8 Packages 2024-02-19T21:51:22.249 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:22.249 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 37 M 2024-02-19T21:51:22.249 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 210 M 2024-02-19T21:51:22.249 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:51:22.319 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:51:22.319 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T21:51:22.319 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T21:51:22.319 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T21:51:22.319 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T21:51:22.319 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T21:51:22.713 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T21:51:22.713 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:22.713 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:51:22.713 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T21:51:22.713 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T21:51:22.714 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T21:51:22.714 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T21:51:22.714 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T21:51:22.714 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T21:51:22.714 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:22.714 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:51:22.808 INFO:teuthology.orchestra.run.smithi173.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 50 MB/s | 8.6 MB 00:00 2024-02-19T21:51:22.836 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T21:51:23.361 INFO:teuthology.orchestra.run.smithi173.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 6.8 MB/s | 3.7 MB 00:00 2024-02-19T21:51:23.498 INFO:teuthology.orchestra.run.smithi173.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2024-02-19T21:51:23.594 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:32 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:51:25.134 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:51:25.134 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-02-19T21:51:25.134 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-02-19T21:51:25.134 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-02-19T21:51:25.134 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout:Install 8 Packages 2024-02-19T21:51:25.135 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:25.136 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 37 M 2024-02-19T21:51:25.136 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 210 M 2024-02-19T21:51:25.136 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:51:25.523 INFO:teuthology.orchestra.run.smithi143.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 59 MB/s | 8.6 MB 00:00 2024-02-19T21:51:26.030 INFO:teuthology.orchestra.run.smithi173.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 1.4 MB/s | 4.7 MB 00:03 2024-02-19T21:51:26.147 INFO:teuthology.orchestra.run.smithi173.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 5.4 MB/s | 645 kB 00:00 2024-02-19T21:51:26.180 INFO:teuthology.orchestra.run.smithi173.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2024-02-19T21:51:26.356 INFO:teuthology.orchestra.run.smithi173.stdout:(7/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 1.3 MB/s | 4.9 MB 00:03 2024-02-19T21:51:26.493 INFO:teuthology.orchestra.run.smithi143.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.8 MB/s | 3.7 MB 00:00 2024-02-19T21:51:26.634 INFO:teuthology.orchestra.run.smithi143.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2024-02-19T21:51:26.664 INFO:teuthology.orchestra.run.smithi173.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 4.5 MB/s | 14 MB 00:03 2024-02-19T21:51:26.664 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:26.665 INFO:teuthology.orchestra.run.smithi173.stdout:Total 8.4 MB/s | 37 MB 00:04 2024-02-19T21:51:26.899 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:51:26.917 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:51:26.917 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:51:27.241 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:51:27.244 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:51:27.866 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:51:28.033 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T21:51:28.180 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T21:51:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T21:51:28.917 INFO:teuthology.orchestra.run.smithi143.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 1.3 MB/s | 4.7 MB 00:03 2024-02-19T21:51:28.972 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T21:51:29.017 INFO:teuthology.orchestra.run.smithi143.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 6.3 MB/s | 645 kB 00:00 2024-02-19T21:51:29.059 INFO:teuthology.orchestra.run.smithi143.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 4.0 MB/s | 172 kB 00:00 2024-02-19T21:51:29.200 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:51:29.243 INFO:teuthology.orchestra.run.smithi143.stdout:(7/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 1.3 MB/s | 4.9 MB 00:03 2024-02-19T21:51:29.796 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:51:29.975 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T21:51:30.652 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T21:51:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T21:51:31.733 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T21:51:33.487 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T21:51:34.143 INFO:teuthology.orchestra.run.smithi143.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 1.9 MB/s | 14 MB 00:07 2024-02-19T21:51:34.144 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:34.144 INFO:teuthology.orchestra.run.smithi143.stdout:Total 4.1 MB/s | 37 MB 00:09 2024-02-19T21:51:34.377 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:51:34.395 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:51:34.396 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:51:34.736 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:51:34.739 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:51:34.824 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T21:51:35.077 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T21:51:35.365 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:51:35.374 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T21:51:35.375 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T21:51:35.375 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T21:51:35.375 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:51:35.375 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T21:51:35.375 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T21:51:35.375 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T21:51:35.375 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T21:51:35.522 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T21:51:35.668 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T21:51:35.780 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T21:51:35.780 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:35.780 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:51:35.780 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T21:51:35.780 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T21:51:35.780 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T21:51:35.780 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T21:51:35.780 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T21:51:35.781 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T21:51:35.781 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T21:51:35.781 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T21:51:35.781 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:35.781 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:51:35.854 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T21:51:35.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-rook 2024-02-19T21:51:36.448 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T21:51:36.655 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:51:36.656 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:49 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:51:37.267 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:51:37.433 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T21:51:38.121 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T21:51:38.201 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:51:38.202 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-19T21:51:38.202 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:38.202 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T21:51:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T21:51:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:38.204 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:51:38.204 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:38.204 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-02-19T21:51:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:38.204 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 1.9 M 2024-02-19T21:51:38.205 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 24 M 2024-02-19T21:51:38.205 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:51:38.458 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T21:51:38.662 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 553 kB/s | 27 kB 00:00 2024-02-19T21:51:38.712 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 202 kB/s | 20 kB 00:00 2024-02-19T21:51:38.737 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 214 kB/s | 27 kB 00:00 2024-02-19T21:51:38.741 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T21:51:38.846 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 823 kB/s | 110 kB 00:00 2024-02-19T21:51:38.871 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.1 MB/s | 155 kB 00:00 2024-02-19T21:51:38.896 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 539 kB/s | 126 kB 00:00 2024-02-19T21:51:38.922 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 563 kB/s | 43 kB 00:00 2024-02-19T21:51:39.039 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 115 kB/s | 16 kB 00:00 2024-02-19T21:51:39.064 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 171 kB/s | 33 kB 00:00 2024-02-19T21:51:39.098 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 463 kB/s | 81 kB 00:00 2024-02-19T21:51:39.190 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 667 kB/s | 61 kB 00:00 2024-02-19T21:51:39.215 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 426 kB/s | 64 kB 00:00 2024-02-19T21:51:39.299 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.5 MB/s | 1.2 MB 00:00 2024-02-19T21:51:39.299 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:39.299 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2024-02-19T21:51:39.353 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:51:39.369 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:51:39.369 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:51:39.455 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:51:39.458 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:51:40.050 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:51:40.297 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T21:51:40.486 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T21:51:40.488 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T21:51:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T21:51:40.997 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T21:51:41.197 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T21:51:41.449 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T21:51:41.765 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T21:51:41.786 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T21:51:42.009 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T21:51:42.020 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T21:51:42.336 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T21:51:42.336 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T21:51:42.336 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T21:51:42.336 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:51:42.337 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T21:51:42.337 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T21:51:42.337 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T21:51:42.337 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T21:51:42.745 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:42.746 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:51:42.863 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-rook 2024-02-19T21:51:43.172 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T21:51:43.488 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T21:51:43.623 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:52 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:51:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T21:51:43.897 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T21:51:44.089 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T21:51:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T21:51:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T21:51:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T21:51:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T21:51:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T21:51:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T21:51:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T21:51:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T21:51:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T21:51:44.380 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T21:51:44.380 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T21:51:44.380 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T21:51:44.380 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T21:51:44.848 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T21:51:44.849 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T21:51:44.850 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T21:51:44.850 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T21:51:44.850 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:44.850 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:51:44.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T21:51:45.167 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:51:45.168 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T21:51:45.169 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T21:51:45.170 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T21:51:45.170 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T21:51:45.170 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T21:51:45.170 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T21:51:45.170 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:45.170 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:51:45.170 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:45.170 INFO:teuthology.orchestra.run.smithi143.stdout:Install 13 Packages 2024-02-19T21:51:45.170 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:45.171 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 1.9 M 2024-02-19T21:51:45.171 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 24 M 2024-02-19T21:51:45.171 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:51:45.626 INFO:teuthology.orchestra.run.smithi143.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 530 kB/s | 27 kB 00:00 2024-02-19T21:51:45.668 INFO:teuthology.orchestra.run.smithi143.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 215 kB/s | 20 kB 00:00 2024-02-19T21:51:45.694 INFO:teuthology.orchestra.run.smithi143.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 225 kB/s | 27 kB 00:00 2024-02-19T21:51:45.732 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:58 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:51:45.736 INFO:teuthology.orchestra.run.smithi143.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2024-02-19T21:51:45.761 INFO:teuthology.orchestra.run.smithi143.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.2 MB/s | 110 kB 00:00 2024-02-19T21:51:45.887 INFO:teuthology.orchestra.run.smithi143.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 284 kB/s | 43 kB 00:00 2024-02-19T21:51:45.945 INFO:teuthology.orchestra.run.smithi143.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 617 kB/s | 155 kB 00:00 2024-02-19T21:51:46.271 INFO:teuthology.orchestra.run.smithi143.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 42 kB/s | 16 kB 00:00 2024-02-19T21:51:46.288 INFO:teuthology.orchestra.run.smithi143.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 63 kB/s | 33 kB 00:00 2024-02-19T21:51:46.729 INFO:teuthology.orchestra.run.smithi143.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 104 kB/s | 81 kB 00:00 2024-02-19T21:51:47.030 INFO:teuthology.orchestra.run.smithi143.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 204 kB/s | 61 kB 00:00 2024-02-19T21:51:47.255 INFO:teuthology.orchestra.run.smithi143.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 66 kB/s | 64 kB 00:00 2024-02-19T21:51:47.294 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:51:47.295 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:47.295 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-02-19T21:51:47.295 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:47.295 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-02-19T21:51:47.296 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:47.297 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 5.9 M 2024-02-19T21:51:47.297 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 24 M 2024-02-19T21:51:47.297 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:51:47.542 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 89 kB 00:00 2024-02-19T21:51:47.750 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 2.1 MB/s | 578 kB 00:00 2024-02-19T21:51:47.876 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 63 kB/s | 21 kB 00:00 2024-02-19T21:51:48.136 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.4 MB/s | 4.8 MB 00:00 2024-02-19T21:51:48.168 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 167 kB/s | 49 kB 00:00 2024-02-19T21:51:48.335 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 609 kB/s | 355 kB 00:00 2024-02-19T21:51:48.335 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:48.335 INFO:teuthology.orchestra.run.smithi173.stdout:Total 5.6 MB/s | 5.9 MB 00:01 2024-02-19T21:51:48.387 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:51:48.396 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:51:48.397 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:51:48.497 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:51:48.500 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:51:48.779 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:51:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T21:51:49.356 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T21:51:49.771 INFO:teuthology.orchestra.run.smithi143.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 341 kB/s | 1.2 MB 00:03 2024-02-19T21:51:49.772 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:49.772 INFO:teuthology.orchestra.run.smithi143.stdout:Total 425 kB/s | 1.9 MB 00:04 2024-02-19T21:51:49.830 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:51:49.845 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:51:49.845 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:51:49.934 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:51:49.937 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:51:50.308 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T21:51:50.454 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:51:50.659 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T21:51:50.675 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T21:51:50.827 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T21:51:50.942 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T21:51:51.041 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T21:51:51.146 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:51:51.294 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T21:51:51.485 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T21:51:51.486 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:51:51.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T21:51:51.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T21:51:51.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T21:51:51.487 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T21:51:51.487 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T21:51:51.712 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T21:51:51.920 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T21:51:51.920 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:51.920 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:51:51.921 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T21:51:51.921 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T21:51:51.921 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T21:51:51.921 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:51.921 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:51:52.046 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T21:51:52.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-02-19T21:51:52.276 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T21:51:52.808 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:05 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:51:53.427 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T21:51:53.701 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T21:51:53.896 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T21:51:54.102 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T21:51:54.244 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T21:51:54.348 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:51:54.348 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:54.348 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-19T21:51:54.349 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:54.349 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:51:54.349 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T21:51:54.349 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-02-19T21:51:54.349 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T21:51:54.349 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:54.349 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:51:54.349 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:54.349 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-02-19T21:51:54.349 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:54.350 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 968 k 2024-02-19T21:51:54.350 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 3.2 M 2024-02-19T21:51:54.350 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:51:54.454 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2024-02-19T21:51:54.479 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 947 kB/s | 83 kB 00:00 2024-02-19T21:51:54.480 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:54.480 INFO:teuthology.orchestra.run.smithi173.stdout:Total 7.2 MB/s | 968 kB 00:00 2024-02-19T21:51:54.490 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:51:54.511 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:51:54.511 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T21:51:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T21:51:54.597 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:51:54.601 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:51:54.818 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:51:55.007 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T21:51:55.008 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:55.008 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:51:55.008 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T21:51:55.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T21:51:55.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T21:51:55.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T21:51:55.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T21:51:55.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T21:51:55.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T21:51:55.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T21:51:55.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T21:51:55.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T21:51:55.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T21:51:55.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T21:51:55.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T21:51:55.009 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:55.009 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:51:55.129 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T21:51:55.136 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T21:51:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T21:51:55.718 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T21:51:55.718 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T21:51:55.892 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:04 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:51:56.088 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T21:51:56.088 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:56.088 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:51:56.088 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T21:51:56.088 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:56.088 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:51:56.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-02-19T21:51:57.081 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:10 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:51:57.443 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:51:57.444 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:57.444 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-02-19T21:51:57.444 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:57.444 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:51:57.444 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T21:51:57.444 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-02-19T21:51:57.444 INFO:teuthology.orchestra.run.smithi143.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T21:51:57.444 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T21:51:57.444 INFO:teuthology.orchestra.run.smithi143.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T21:51:57.444 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T21:51:57.444 INFO:teuthology.orchestra.run.smithi143.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T21:51:57.445 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:57.445 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:51:57.445 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:51:57.445 INFO:teuthology.orchestra.run.smithi143.stdout:Install 6 Packages 2024-02-19T21:51:57.445 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:51:57.445 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 5.9 M 2024-02-19T21:51:57.446 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 24 M 2024-02-19T21:51:57.446 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:51:57.716 INFO:teuthology.orchestra.run.smithi143.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2024-02-19T21:51:57.958 INFO:teuthology.orchestra.run.smithi143.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 88 kB/s | 21 kB 00:00 2024-02-19T21:51:58.206 INFO:teuthology.orchestra.run.smithi143.stdout:(3/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.1 MB/s | 578 kB 00:00 2024-02-19T21:51:58.265 INFO:teuthology.orchestra.run.smithi143.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.1 MB/s | 355 kB 00:00 2024-02-19T21:51:58.558 INFO:teuthology.orchestra.run.smithi143.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 139 kB/s | 49 kB 00:00 2024-02-19T21:51:58.638 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:51:58.639 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-19T21:51:58.639 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:58.639 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:51:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T21:51:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:58.639 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:51:58.640 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:51:58.640 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-19T21:51:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:58.640 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 131 k 2024-02-19T21:51:58.640 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 439 k 2024-02-19T21:51:58.641 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:51:58.692 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.5 MB/s | 131 kB 00:00 2024-02-19T21:51:58.692 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:58.692 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.4 MB/s | 131 kB 00:00 2024-02-19T21:51:58.693 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:51:58.698 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:51:58.698 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:51:58.747 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:51:58.750 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:51:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:51:59.102 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:51:59.437 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:51:59.820 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:51:59.820 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:59.820 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:51:59.820 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T21:51:59.820 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:51:59.821 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:00.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-02-19T21:52:00.609 INFO:teuthology.orchestra.run.smithi143.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 1.6 MB/s | 4.8 MB 00:02 2024-02-19T21:52:00.611 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:00.611 INFO:teuthology.orchestra.run.smithi143.stdout:Total 1.8 MB/s | 5.9 MB 00:03 2024-02-19T21:52:00.667 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:52:00.677 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:52:00.677 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:52:00.783 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:52:00.786 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:52:00.842 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:13 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:52:01.066 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:52:01.341 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T21:52:01.638 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T21:52:02.358 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:02.406 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:52:02.407 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-19T21:52:02.407 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:02.600 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T21:52:02.949 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T21:52:03.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-02-19T21:52:03.214 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T21:52:03.442 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:52:03.736 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:52:03.736 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T21:52:03.736 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T21:52:03.736 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T21:52:03.736 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T21:52:03.736 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T21:52:03.891 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:16 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:52:04.131 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T21:52:04.131 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:04.131 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:52:04.131 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T21:52:04.131 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T21:52:04.131 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T21:52:04.131 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:04.131 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:04.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-fuse 2024-02-19T21:52:05.019 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:14 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:05.435 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:52:05.436 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:05.436 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:05.436 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:05.436 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:52:05.436 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T21:52:05.436 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:05.436 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:52:05.436 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:05.436 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-19T21:52:05.436 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:05.437 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 24 k 2024-02-19T21:52:05.437 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 94 k 2024-02-19T21:52:05.437 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:52:05.489 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 462 kB/s | 24 kB 00:00 2024-02-19T21:52:05.490 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:05.491 INFO:teuthology.orchestra.run.smithi173.stdout:Total 452 kB/s | 24 kB 00:00 2024-02-19T21:52:05.491 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:52:05.494 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:52:05.494 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:52:05.496 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:52:05.499 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:52:05.551 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:52:05.806 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:06.075 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:06.428 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:06.428 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:06.428 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:52:06.428 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:06.428 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:06.428 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:06.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-02-19T21:52:06.565 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:06.565 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:06.566 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:06.566 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:06.566 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:52:06.566 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T21:52:06.566 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-02-19T21:52:06.566 INFO:teuthology.orchestra.run.smithi143.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T21:52:06.566 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:06.566 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:52:06.566 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:06.566 INFO:teuthology.orchestra.run.smithi143.stdout:Install 2 Packages 2024-02-19T21:52:06.566 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:06.567 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 968 k 2024-02-19T21:52:06.567 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 3.2 M 2024-02-19T21:52:06.567 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:52:06.860 INFO:teuthology.orchestra.run.smithi143.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-02-19T21:52:06.960 INFO:teuthology.orchestra.run.smithi143.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 522 kB/s | 83 kB 00:00 2024-02-19T21:52:06.961 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:06.961 INFO:teuthology.orchestra.run.smithi143.stdout:Total 2.4 MB/s | 968 kB 00:00 2024-02-19T21:52:06.971 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:52:06.992 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:52:06.992 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:52:07.079 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:52:07.082 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:52:07.253 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:52:07.290 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:20 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:52:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T21:52:07.700 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T21:52:08.081 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T21:52:08.081 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T21:52:08.496 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T21:52:08.496 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:08.497 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:52:08.497 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T21:52:08.497 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:08.497 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:08.791 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:08.837 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:52:08.838 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-19T21:52:08.838 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:09.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-02-19T21:52:09.911 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librados-devel 2024-02-19T21:52:09.934 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:22 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:52:10.684 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:19 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:11.446 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:11.493 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:52:11.495 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-19T21:52:11.495 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:11.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-02-19T21:52:12.257 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:12.257 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:12.257 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:12.258 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:12.258 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:52:12.258 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T21:52:12.258 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:12.258 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:52:12.258 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:12.258 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-02-19T21:52:12.258 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:12.258 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 131 k 2024-02-19T21:52:12.258 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 439 k 2024-02-19T21:52:12.258 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:52:12.294 INFO:teuthology.orchestra.run.smithi143.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 131 kB 00:00 2024-02-19T21:52:12.295 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:12.295 INFO:teuthology.orchestra.run.smithi143.stdout:Total 3.5 MB/s | 131 kB 00:00 2024-02-19T21:52:12.295 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:52:12.300 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:52:12.300 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:52:12.346 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:25 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:52:12.351 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:52:12.354 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:52:12.496 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:52:12.731 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:13.032 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:13.402 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:13.402 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:13.402 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:52:13.403 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:13.403 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:13.403 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:13.659 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install libcephfs2 2024-02-19T21:52:13.836 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:13.882 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:52:13.884 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-19T21:52:13.884 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:13.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-02-19T21:52:14.431 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:23 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:14.741 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:27 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:52:15.927 INFO:teuthology.orchestra.run.smithi143.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:15.974 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:15.976 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-02-19T21:52:15.976 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:16.244 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:16.290 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:52:16.292 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-19T21:52:16.292 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:16.312 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install libcephfs-devel 2024-02-19T21:52:16.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-02-19T21:52:17.109 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:26 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:17.144 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:30 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:52:18.633 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:18.675 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:18.676 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:52:18.676 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T21:52:18.676 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:52:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:18.676 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-02-19T21:52:18.676 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 24 k 2024-02-19T21:52:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 94 k 2024-02-19T21:52:18.677 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:52:18.679 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:52:18.680 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-19T21:52:18.680 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 305 kB/s | 24 kB 00:00 2024-02-19T21:52:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:Total 301 kB/s | 24 kB 00:00 2024-02-19T21:52:18.760 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:52:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:52:18.761 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:52:18.763 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:52:18.766 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:52:18.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-02-19T21:52:18.820 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:52:19.025 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:19.279 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:19.529 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:32 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:52:19.637 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:19.637 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:19.637 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:52:19.637 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:19.637 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:19.637 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:19.759 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librados2 2024-02-19T21:52:20.524 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:29 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:21.004 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:21.051 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:52:21.052 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-02-19T21:52:21.052 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:21.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-02-19T21:52:21.907 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:34 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:52:22.026 INFO:teuthology.orchestra.run.smithi143.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:22.072 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:22.073 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-02-19T21:52:22.073 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:22.184 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librbd1 2024-02-19T21:52:22.947 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:31 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:23.467 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:52:23.467 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:23.467 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:23.467 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:23.467 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:52:23.468 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T21:52:23.468 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:23.468 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:52:23.468 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:23.468 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-19T21:52:23.468 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:23.469 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 89 k 2024-02-19T21:52:23.469 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 298 k 2024-02-19T21:52:23.469 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:52:23.511 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2024-02-19T21:52:23.512 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:23.512 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.0 MB/s | 89 kB 00:00 2024-02-19T21:52:23.512 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:52:23.522 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:52:23.522 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:52:23.570 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:52:23.573 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:52:23.677 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:52:23.873 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:24.166 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:24.444 INFO:teuthology.orchestra.run.smithi143.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:24.491 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:24.492 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-02-19T21:52:24.492 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:24.541 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:24.541 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:24.542 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:52:24.542 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:24.542 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:24.542 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:24.610 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rados 2024-02-19T21:52:24.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-02-19T21:52:25.365 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:34 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:25.560 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:38 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:52:26.875 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:26.921 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:26.923 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-02-19T21:52:26.923 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:27.042 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rgw 2024-02-19T21:52:27.111 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:52:27.111 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:27.111 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:27.111 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:52:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T21:52:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:27.112 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:52:27.112 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:27.112 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-19T21:52:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:27.112 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.7 M 2024-02-19T21:52:27.113 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 18 M 2024-02-19T21:52:27.113 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:52:27.232 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 31 MB/s | 3.7 MB 00:00 2024-02-19T21:52:27.233 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:27.233 INFO:teuthology.orchestra.run.smithi173.stdout:Total 31 MB/s | 3.7 MB 00:00 2024-02-19T21:52:27.233 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:52:27.245 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:52:27.245 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:52:27.308 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:52:27.311 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:52:27.778 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:52:27.802 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:36 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:28.007 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:28.424 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:28.797 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:28.798 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:28.798 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:52:28.798 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:28.798 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:28.798 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:29.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-02-19T21:52:29.320 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:29.368 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:29.369 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-02-19T21:52:29.369 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:29.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-cephfs 2024-02-19T21:52:29.820 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:42 ago on Mon 19 Feb 2024 09:49:47 PM UTC. 2024-02-19T21:52:30.239 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:39 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:31.369 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-02-19T21:52:31.370 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:31.370 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:31.370 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:31.370 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-02-19T21:52:31.370 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T21:52:31.370 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:31.370 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-02-19T21:52:31.370 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-02-19T21:52:31.370 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-02-19T21:52:31.370 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:31.371 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 173 k 2024-02-19T21:52:31.371 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 649 k 2024-02-19T21:52:31.371 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-02-19T21:52:31.418 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 173 kB 00:00 2024-02-19T21:52:31.419 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:31.419 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.5 MB/s | 173 kB 00:00 2024-02-19T21:52:31.419 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-02-19T21:52:31.430 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-02-19T21:52:31.430 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-02-19T21:52:31.481 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-02-19T21:52:31.484 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-02-19T21:52:31.594 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-02-19T21:52:31.746 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:31.792 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:31.793 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-02-19T21:52:31.793 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:31.835 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:31.889 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rbd 2024-02-19T21:52:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:32.545 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:32.545 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:32.545 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-02-19T21:52:32.545 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:32.545 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:52:32.545 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-02-19T21:52:32.651 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:41 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:32.766 DEBUG:teuthology.parallel:result is None 2024-02-19T21:52:34.142 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:52:34.188 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:34.190 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-02-19T21:52:34.190 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:34.333 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-fuse 2024-02-19T21:52:35.092 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:44 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:36.649 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:36.649 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:36.649 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:36.649 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:36.649 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:52:36.649 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T21:52:36.650 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:36.650 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:52:36.650 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:36.650 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-02-19T21:52:36.650 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:36.650 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 89 k 2024-02-19T21:52:36.651 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 298 k 2024-02-19T21:52:36.651 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:52:36.697 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 89 kB 00:00 2024-02-19T21:52:36.697 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:36.698 INFO:teuthology.orchestra.run.smithi143.stdout:Total 1.8 MB/s | 89 kB 00:00 2024-02-19T21:52:36.698 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:52:36.708 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:52:36.708 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:52:36.758 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:52:36.762 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:52:36.870 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:52:37.103 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:37.580 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:37.975 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:37.976 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:37.976 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:52:37.976 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:37.976 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:37.976 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:38.166 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-mirror 2024-02-19T21:52:38.933 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:47 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:40.491 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:40.492 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:40.492 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:40.492 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:40.493 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:52:40.493 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T21:52:40.493 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:40.493 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:52:40.493 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:40.493 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-02-19T21:52:40.493 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:40.493 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 3.7 M 2024-02-19T21:52:40.493 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 18 M 2024-02-19T21:52:40.493 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:52:40.618 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2024-02-19T21:52:40.619 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:40.619 INFO:teuthology.orchestra.run.smithi143.stdout:Total 29 MB/s | 3.7 MB 00:00 2024-02-19T21:52:40.620 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:52:40.631 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:52:40.631 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:52:40.696 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:52:40.699 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:52:41.162 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:52:41.386 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:41.833 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:42.218 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:42.219 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:42.219 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:52:42.219 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:42.219 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:42.219 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:42.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-nbd 2024-02-19T21:52:43.231 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:02:52 ago on Mon 19 Feb 2024 09:49:51 PM UTC. 2024-02-19T21:52:44.783 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-02-19T21:52:44.784 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:44.784 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:44.784 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:44.784 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-02-19T21:52:44.784 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T21:52:44.784 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:44.784 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-02-19T21:52:44.784 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-02-19T21:52:44.784 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-02-19T21:52:44.784 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:44.785 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 173 k 2024-02-19T21:52:44.785 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 649 k 2024-02-19T21:52:44.785 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-02-19T21:52:44.852 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.5 MB/s | 173 kB 00:00 2024-02-19T21:52:44.853 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:44.853 INFO:teuthology.orchestra.run.smithi143.stdout:Total 2.5 MB/s | 173 kB 00:00 2024-02-19T21:52:44.853 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-02-19T21:52:44.863 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-02-19T21:52:44.863 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-02-19T21:52:44.912 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-02-19T21:52:44.915 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-02-19T21:52:45.036 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-02-19T21:52:45.273 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:45.550 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:45.975 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:52:45.975 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:45.975 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-02-19T21:52:45.975 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.975 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-02-19T21:52:45.975 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-02-19T21:52:46.210 DEBUG:teuthology.parallel:result is None 2024-02-19T21:52:46.210 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:52:46.210 INFO:teuthology.packaging:ref: None 2024-02-19T21:52:46.210 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:52:46.210 INFO:teuthology.packaging:branch: None 2024-02-19T21:52:46.210 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:52:46.211 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T21:52:46.356 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T21:52:46.378 INFO:teuthology.orchestra.run.smithi143.stdout:16.2.4-0.el8 2024-02-19T21:52:46.379 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T21:52:46.379 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T21:52:46.381 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:52:46.381 INFO:teuthology.packaging:ref: None 2024-02-19T21:52:46.381 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:52:46.381 INFO:teuthology.packaging:branch: None 2024-02-19T21:52:46.381 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:52:46.381 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T21:52:46.617 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T21:52:46.644 INFO:teuthology.orchestra.run.smithi173.stdout:16.2.4-0.el8 2024-02-19T21:52:46.644 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T21:52:46.644 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T21:52:46.646 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T21:52:46.646 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-02-19T21:52:46.646 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T21:52:46.675 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-19T21:52:46.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T21:52:46.717 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T21:52:46.717 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-02-19T21:52:46.718 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T21:52:46.745 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T21:52:46.812 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-19T21:52:46.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T21:52:46.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T21:52:46.913 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T21:52:46.914 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-02-19T21:52:46.914 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T21:52:46.940 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T21:52:47.007 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-19T21:52:47.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T21:52:47.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T21:52:47.105 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T21:52:47.105 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-02-19T21:52:47.105 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T21:52:47.132 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T21:52:47.200 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-19T21:52:47.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T21:52:47.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T21:52:47.296 INFO:teuthology.run_tasks:Running task print... 2024-02-19T21:52:47.307 INFO:teuthology.task.print:**** done install task... 2024-02-19T21:52:47.307 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-19T21:52:47.443 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'Degraded data redundancy', '\\(PG_', 'MDS_INSUFFICIENT_STANDBY', 'deprecated feature inline_data', 'compat changed unexpectedly', 'overall HEALTH_', 'OSD_DOWN', 'OSD_', 'but it is still running', 'is not responding'], 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T21:52:47.444 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-19T21:52:47.444 INFO:tasks.cephadm:Cluster fsid is 38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:52:47.444 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-19T21:52:47.444 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-19T21:52:47.444 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi143': '172.21.15.143', 'mon.smithi173': '172.21.15.173'} 2024-02-19T21:52:47.444 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-19T21:52:47.445 DEBUG:teuthology.orchestra.run.smithi143:> sudo hostname $(hostname -s) 2024-02-19T21:52:47.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-02-19T21:52:47.501 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-19T21:52:47.502 DEBUG:teuthology.orchestra.run.smithi143:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T21:52:47.663 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 21:52 /home/ubuntu/cephtest/cephadm 2024-02-19T21:52:47.664 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T21:52:47.798 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 21:52 /home/ubuntu/cephtest/cephadm 2024-02-19T21:52:47.799 DEBUG:teuthology.orchestra.run.smithi143:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T21:52:47.818 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T21:52:47.847 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph 2024-02-19T21:52:47.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-02-19T21:52:47.913 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /etc/ceph 2024-02-19T21:52:47.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-02-19T21:52:47.980 INFO:tasks.cephadm:Writing seed config... 2024-02-19T21:52:47.981 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-19T21:52:47.981 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-19T21:52:47.982 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-19T21:52:47.983 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-19T21:52:47.984 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-19T21:52:47.985 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-02-19T21:52:47.985 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-19T21:52:48.010 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 38187086-cf71-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-19T21:52:48.010 DEBUG:teuthology.orchestra.run.smithi143:mon.smithi143> sudo journalctl -f -n 0 -u ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143.service 2024-02-19T21:52:48.052 INFO:tasks.cephadm:Bootstrapping... 2024-02-19T21:52:48.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 38187086-cf71-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.143 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-19T21:52:48.080 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:-- Logs begin at Mon 2024-02-19 21:42:40 UTC. -- 2024-02-19T21:52:48.319 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T21:52:49.274 INFO:teuthology.orchestra.run.smithi143.stderr:/bin/podman: 4.9.3-dev 2024-02-19T21:52:49.285 INFO:teuthology.orchestra.run.smithi143.stderr:Verifying podman|docker is present... 2024-02-19T21:52:49.285 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T21:52:49.307 INFO:teuthology.orchestra.run.smithi143.stderr:/bin/podman: 4.9.3-dev 2024-02-19T21:52:49.320 INFO:teuthology.orchestra.run.smithi143.stderr:Verifying lvm2 is present... 2024-02-19T21:52:49.320 INFO:teuthology.orchestra.run.smithi143.stderr:Verifying time synchronization is in place... 2024-02-19T21:52:49.320 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T21:52:49.327 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T21:52:49.328 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-active chrony.service 2024-02-19T21:52:49.334 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: inactive 2024-02-19T21:52:49.334 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T21:52:49.340 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: enabled 2024-02-19T21:52:49.341 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T21:52:49.346 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: active 2024-02-19T21:52:49.347 INFO:teuthology.orchestra.run.smithi143.stderr:Unit chronyd.service is enabled and running 2024-02-19T21:52:49.347 INFO:teuthology.orchestra.run.smithi143.stderr:Repeating the final host check... 2024-02-19T21:52:49.347 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T21:52:49.367 INFO:teuthology.orchestra.run.smithi143.stderr:/bin/podman: 4.9.3-dev 2024-02-19T21:52:49.382 INFO:teuthology.orchestra.run.smithi143.stderr:podman|docker (/bin/podman) is present 2024-02-19T21:52:49.382 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl is present 2024-02-19T21:52:49.382 INFO:teuthology.orchestra.run.smithi143.stderr:lvcreate is present 2024-02-19T21:52:49.383 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T21:52:49.388 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T21:52:49.389 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-active chrony.service 2024-02-19T21:52:49.394 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: inactive 2024-02-19T21:52:49.395 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T21:52:49.400 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: enabled 2024-02-19T21:52:49.401 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T21:52:49.406 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: active 2024-02-19T21:52:49.407 INFO:teuthology.orchestra.run.smithi143.stderr:Unit chronyd.service is enabled and running 2024-02-19T21:52:49.407 INFO:teuthology.orchestra.run.smithi143.stderr:Host looks OK 2024-02-19T21:52:49.407 INFO:teuthology.orchestra.run.smithi143.stderr:Cluster fsid: 38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:52:49.407 INFO:teuthology.orchestra.run.smithi143.stderr:Acquiring lock 139927742573032 on /run/cephadm/38187086-cf71-11ee-95bb-87774f69a715.lock 2024-02-19T21:52:49.408 INFO:teuthology.orchestra.run.smithi143.stderr:Lock 139927742573032 acquired on /run/cephadm/38187086-cf71-11ee-95bb-87774f69a715.lock 2024-02-19T21:52:49.408 INFO:teuthology.orchestra.run.smithi143.stderr:Verifying IP 172.21.15.143 port 3300 ... 2024-02-19T21:52:49.408 INFO:teuthology.orchestra.run.smithi143.stderr:Verifying IP 172.21.15.143 port 6789 ... 2024-02-19T21:52:49.408 INFO:teuthology.orchestra.run.smithi143.stderr:Base mon IP is 172.21.15.143, final addrv is [v2:172.21.15.143:3300,v1:172.21.15.143:6789] 2024-02-19T21:52:49.408 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /sbin/ip route ls 2024-02-19T21:52:49.411 INFO:teuthology.orchestra.run.smithi143.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.143 metric 100 2024-02-19T21:52:49.411 INFO:teuthology.orchestra.run.smithi143.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.143 metric 100 2024-02-19T21:52:49.411 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /sbin/ip -6 route ls 2024-02-19T21:52:49.414 INFO:teuthology.orchestra.run.smithi143.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-19T21:52:49.414 INFO:teuthology.orchestra.run.smithi143.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-19T21:52:49.414 INFO:teuthology.orchestra.run.smithi143.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1763sec hoplimit 64 pref medium 2024-02-19T21:52:49.414 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /sbin/ip -6 addr ls 2024-02-19T21:52:49.416 INFO:teuthology.orchestra.run.smithi143.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-19T21:52:49.417 INFO:teuthology.orchestra.run.smithi143.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-19T21:52:49.417 INFO:teuthology.orchestra.run.smithi143.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T21:52:49.417 INFO:teuthology.orchestra.run.smithi143.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-19T21:52:49.417 INFO:teuthology.orchestra.run.smithi143.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7783/64 scope link 2024-02-19T21:52:49.417 INFO:teuthology.orchestra.run.smithi143.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T21:52:49.417 INFO:teuthology.orchestra.run.smithi143.stderr:Mon IP 172.21.15.143 is in CIDR network 172.21.0.0/20 2024-02-19T21:52:49.418 INFO:teuthology.orchestra.run.smithi143.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-19T21:52:49.419 INFO:teuthology.orchestra.run.smithi143.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-19T21:52:49.419 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-19T21:52:49.516 INFO:teuthology.orchestra.run.smithi143.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-19T21:52:56.144 INFO:teuthology.orchestra.run.smithi143.stderr:/bin/podman: Getting image source signatures 2024-02-19T21:52:56.148 INFO:teuthology.orchestra.run.smithi143.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-19T21:52:56.148 INFO:teuthology.orchestra.run.smithi143.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-19T21:53:08.281 INFO:teuthology.orchestra.run.smithi143.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T21:53:08.439 INFO:teuthology.orchestra.run.smithi143.stderr:/bin/podman: Writing manifest to image destination 2024-02-19T21:53:08.699 INFO:teuthology.orchestra.run.smithi143.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T21:53:08.709 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-19T21:53:09.351 INFO:teuthology.orchestra.run.smithi143.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T21:53:09.966 INFO:teuthology.orchestra.run.smithi143.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T21:53:09.966 INFO:teuthology.orchestra.run.smithi143.stderr:Extracting ceph user uid/gid from container image... 2024-02-19T21:53:09.967 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-19T21:53:10.470 INFO:teuthology.orchestra.run.smithi143.stderr:stat: 167 167 2024-02-19T21:53:11.071 INFO:teuthology.orchestra.run.smithi143.stderr:Creating initial keys... 2024-02-19T21:53:11.071 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T21:53:11.674 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-authtool: AQDHzdNlmW0WIBAAKq8WMcY72g3N1A3imA+3ZA== 2024-02-19T21:53:12.144 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T21:53:12.690 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-authtool: AQDIzdNlzJdwIxAAdGArxtZTNVlZ/gVbHqi6YA== 2024-02-19T21:53:13.254 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T21:53:13.789 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-authtool: AQDJzdNl+M7ZJxAAHbgevQeAzHTgfVGoH9tFDw== 2024-02-19T21:53:14.434 INFO:teuthology.orchestra.run.smithi143.stderr:Creating initial monmap... 2024-02-19T21:53:14.435 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp_9pmej_l:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 38187086-cf71-11ee-95bb-87774f69a715 --addv smithi143 [v2:172.21.15.143:3300,v1:172.21.15.143:6789] /tmp/monmap 2024-02-19T21:53:14.920 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T21:53:14.920 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:53:14.920 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T21:53:15.483 INFO:teuthology.orchestra.run.smithi143.stderr:monmaptool for smithi143 [v2:172.21.15.143:3300,v1:172.21.15.143:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T21:53:15.483 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/monmaptool: set fsid to 38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:53:15.483 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T21:53:15.483 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-02-19T21:53:15.483 INFO:teuthology.orchestra.run.smithi143.stderr:Creating mon... 2024-02-19T21:53:15.483 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/mon.smithi143:/var/lib/ceph/mon/ceph-smithi143:z -v /tmp/ceph-tmpnbf2sze9:/tmp/keyring:z -v /tmp/ceph-tmp_9pmej_l:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi143 --fsid 38187086-cf71-11ee-95bb-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-19T21:53:16.093 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.962+0000 7f536364a700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T21:53:16.094 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.962+0000 7f536364a700 1 imported monmap: 2024-02-19T21:53:16.094 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-19T21:53:16.094 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: fsid 38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:53:16.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: last_changed 2024-02-19T21:53:14.854843+0000 2024-02-19T21:53:16.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: created 2024-02-19T21:53:14.854843+0000 2024-02-19T21:53:16.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-19T21:53:16.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-19T21:53:16.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.smithi143 2024-02-19T21:53:16.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.962+0000 7f536364a700 0 /usr/bin/ceph-mon: set fsid to 38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Compile date May 13 2021 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: DB SUMMARY 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi143/store.db dir, Total Num: 0, files: 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi143/store.db: 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.create_if_missing: 1 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T21:53:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.env: 0x5573ff9261c0 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.fs: Posix File System 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.info_log: 0x55740053eec0 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.statistics: (nil) 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.use_fsync: 0 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T21:53:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.db_log_dir: 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi143/store.db 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T21:53:16.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.write_buffer_manager: 0x557400545530 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.unordered_write: 0 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T21:53:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.row_cache: None 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.wal_filter: None 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T21:53:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.max_open_files: -1 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Compression algorithms supported: 2024-02-19T21:53:16.125 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: kZSTD supported: 0 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.963+0000 7f536364a700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi143/store.db/MANIFEST-000001 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.126 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.merge_operator: 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_filter: None 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557400457cb8) 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-19T21:53:16.127 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: block_cache: 0x55740048cd10 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-19T21:53:16.128 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compression: NoCompression 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T21:53:16.129 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.num_levels: 7 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T21:53:16.130 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T21:53:16.131 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T21:53:16.132 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.ttl: 2592000 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi143/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.964+0000 7f536364a700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-19T21:53:16.133 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.965+0000 7f536364a700 4 rocksdb: DB pointer 0x557400551800 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.965+0000 7f534c4e3700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.965+0000 7f534c4e3700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T21:53:16.134 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.135 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T21:53:16.136 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T21:53:16.137 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T21:53:16.137 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T21:53:16.137 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:53:16.137 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:53:16.137 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T21:53:16.137 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.137 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T21:53:16.137 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: 2024-02-19T21:53:16.137 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.966+0000 7f536364a700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-19T21:53:16.137 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.966+0000 7f536364a700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-19T21:53:16.137 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:53:15.966+0000 7f536364a700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi143 for mon.smithi143 2024-02-19T21:53:16.630 INFO:teuthology.orchestra.run.smithi143.stderr:create mon.smithi143 on 2024-02-19T21:53:16.630 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:53:16.635 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl disable ceph.target 2024-02-19T21:53:16.642 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-19T21:53:16.727 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl enable ceph.target 2024-02-19T21:53:16.735 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-19T21:53:16.813 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl start ceph.target 2024-02-19T21:53:16.820 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl enable ceph-38187086-cf71-11ee-95bb-87774f69a715.target 2024-02-19T21:53:16.825 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-38187086-cf71-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-38187086-cf71-11ee-95bb-87774f69a715.target. 2024-02-19T21:53:16.825 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-38187086-cf71-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-38187086-cf71-11ee-95bb-87774f69a715.target. 2024-02-19T21:53:16.904 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl start ceph-38187086-cf71-11ee-95bb-87774f69a715.target 2024-02-19T21:53:16.911 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl daemon-reload 2024-02-19T21:53:16.997 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl stop ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143 2024-02-19T21:53:17.020 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl reset-failed ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143 2024-02-19T21:53:17.026 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: Failed to reset failed state of unit ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143.service: Unit ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143.service not loaded. 2024-02-19T21:53:17.026 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl enable ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143 2024-02-19T21:53:17.033 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: Created symlink /etc/systemd/system/ceph-38187086-cf71-11ee-95bb-87774f69a715.target.wants/ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143.service → /etc/systemd/system/ceph-38187086-cf71-11ee-95bb-87774f69a715@.service. 2024-02-19T21:53:17.117 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl start ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143 2024-02-19T21:53:17.703 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T21:53:17.712 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: disabled 2024-02-19T21:53:17.712 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T21:53:17.719 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: inactive 2024-02-19T21:53:17.720 INFO:teuthology.orchestra.run.smithi143.stderr:firewalld.service is not enabled 2024-02-19T21:53:17.720 INFO:teuthology.orchestra.run.smithi143.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T21:53:17.720 INFO:teuthology.orchestra.run.smithi143.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T21:53:17.721 INFO:teuthology.orchestra.run.smithi143.stderr:Waiting for mon to start... 2024-02-19T21:53:17.721 INFO:teuthology.orchestra.run.smithi143.stderr:Waiting for mon... 2024-02-19T21:53:17.721 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/mon.smithi143:/var/lib/ceph/mon/ceph-smithi143:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-19T21:53:17.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:17 smithi143 bash[37652]: 1b892b1345a12d28f2c2e215e679f48ea168ff99a8ee15bb5d4886965d0ffb93 2024-02-19T21:53:17.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:17 smithi143 systemd[1]: Started Ceph mon.smithi143 for 38187086-cf71-11ee-95bb-87774f69a715. 2024-02-19T21:53:18.301 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.300+0000 7f16247ac700 1 Processor -- start 2024-02-19T21:53:18.301 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.300+0000 7f16247ac700 1 -- start start 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.300+0000 7f16247ac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.300+0000 7f16247ac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c0fc880 con 0x7f161c0703f0 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.301+0000 7f1622548700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.301+0000 7f1622548700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:42750/0 (socket says 172.21.15.143:42750) 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.301+0000 7f1622548700 1 -- 172.21.15.143:0/193204192 learned_addr learned my addr 172.21.15.143:0/193204192 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.301+0000 7f1622548700 1 -- 172.21.15.143:0/193204192 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161c101380 con 0x7f161c0703f0 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.301+0000 7f1622548700 1 --2- 172.21.15.143:0/193204192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c100e90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f160c00b5f0 tx=0x7f160c006ee0).ready entity=mon.0 client_cookie=f486e0d934101855 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.301+0000 7f1621546700 1 -- 172.21.15.143:0/193204192 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f160c00d600 con 0x7f161c0703f0 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.301+0000 7f1621546700 1 -- 172.21.15.143:0/193204192 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c00d760 con 0x7f161c0703f0 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.301+0000 7f1621546700 1 -- 172.21.15.143:0/193204192 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f160c00da30 con 0x7f161c0703f0 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.302+0000 7f16247ac700 1 -- 172.21.15.143:0/193204192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 msgr2=0x7f161c100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:18.302 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.302+0000 7f16247ac700 1 --2- 172.21.15.143:0/193204192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c100e90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f160c00b5f0 tx=0x7f160c006ee0).stop 2024-02-19T21:53:18.304 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.302+0000 7f16247ac700 1 -- 172.21.15.143:0/193204192 shutdown_connections 2024-02-19T21:53:18.304 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.302+0000 7f16247ac700 1 --2- 172.21.15.143:0/193204192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c100e90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:18.304 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.302+0000 7f16247ac700 1 -- 172.21.15.143:0/193204192 >> 172.21.15.143:0/193204192 conn(0x7f161c0f6530 msgr2=0x7f161c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:18.304 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.302+0000 7f16247ac700 1 -- 172.21.15.143:0/193204192 shutdown_connections 2024-02-19T21:53:18.304 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.302+0000 7f16247ac700 1 -- 172.21.15.143:0/193204192 wait complete. 2024-02-19T21:53:18.304 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.303+0000 7f16247ac700 1 Processor -- start 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.303+0000 7f16247ac700 1 -- start start 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.303+0000 7f16247ac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c10c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.303+0000 7f16247ac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c10ca80 con 0x7f161c0703f0 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.303+0000 7f1622548700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c10c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.303+0000 7f1622548700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c10c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:42762/0 (socket says 172.21.15.143:42762) 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.303+0000 7f1622548700 1 -- 172.21.15.143:0/4177163141 learned_addr learned my addr 172.21.15.143:0/4177163141 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.303+0000 7f1622548700 1 -- 172.21.15.143:0/4177163141 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c00b200 con 0x7f161c0703f0 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.304+0000 7f1622548700 1 --2- 172.21.15.143:0/4177163141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c10c590 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f160c006c70 tx=0x7f160c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.304+0000 7f160affd700 1 -- 172.21.15.143:0/4177163141 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f160c011c80 con 0x7f161c0703f0 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.304+0000 7f160affd700 1 -- 172.21.15.143:0/4177163141 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c011de0 con 0x7f161c0703f0 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.304+0000 7f16247ac700 1 -- 172.21.15.143:0/4177163141 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161c10cc80 con 0x7f161c0703f0 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.304+0000 7f160affd700 1 -- 172.21.15.143:0/4177163141 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f160c01d5d0 con 0x7f161c0703f0 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.304+0000 7f16247ac700 1 -- 172.21.15.143:0/4177163141 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161c10d110 con 0x7f161c0703f0 2024-02-19T21:53:18.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.304+0000 7f160affd700 1 -- 172.21.15.143:0/4177163141 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f160c00f070 con 0x7f161c0703f0 2024-02-19T21:53:18.306 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.304+0000 7f160affd700 1 -- 172.21.15.143:0/4177163141 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f160c027460 con 0x7f161c0703f0 2024-02-19T21:53:18.306 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.305+0000 7f16247ac700 1 -- 172.21.15.143:0/4177163141 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161c045ad0 con 0x7f161c0703f0 2024-02-19T21:53:18.306 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.307+0000 7f160affd700 1 -- 172.21.15.143:0/4177163141 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f160c016070 con 0x7f161c0703f0 2024-02-19T21:53:18.375 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.373+0000 7f16247ac700 1 -- 172.21.15.143:0/4177163141 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f161c1062a0 con 0x7f161c0703f0 2024-02-19T21:53:18.375 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.374+0000 7f160affd700 1 -- 172.21.15.143:0/4177163141 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f160c0276a0 con 0x7f161c0703f0 2024-02-19T21:53:18.375 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: cluster: 2024-02-19T21:53:18.375 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: id: 38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:53:18.375 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-19T21:53:18.375 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.375 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: services: 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi143 (age 0.722111s) 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: data: 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: pgs: 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.374+0000 7f16247ac700 1 -- 172.21.15.143:0/4177163141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 msgr2=0x7f161c10c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.374+0000 7f16247ac700 1 --2- 172.21.15.143:0/4177163141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c10c590 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f160c006c70 tx=0x7f160c016000).stop 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.374+0000 7f16247ac700 1 -- 172.21.15.143:0/4177163141 shutdown_connections 2024-02-19T21:53:18.376 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.374+0000 7f16247ac700 1 --2- 172.21.15.143:0/4177163141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f161c0703f0 0x7f161c10c590 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:18.377 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.374+0000 7f16247ac700 1 -- 172.21.15.143:0/4177163141 >> 172.21.15.143:0/4177163141 conn(0x7f161c0f6530 msgr2=0x7f161c0f8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:18.377 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.374+0000 7f16247ac700 1 -- 172.21.15.143:0/4177163141 shutdown_connections 2024-02-19T21:53:18.377 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:18.374+0000 7f16247ac700 1 -- 172.21.15.143:0/4177163141 wait complete. 2024-02-19T21:53:18.915 INFO:teuthology.orchestra.run.smithi143.stderr:mon is available 2024-02-19T21:53:18.916 INFO:teuthology.orchestra.run.smithi143.stderr:Assimilating anything we can from ceph.conf... 2024-02-19T21:53:18.916 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z -v /var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/mon.smithi143:/var/lib/ceph/mon/ceph-smithi143:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi143/config 2024-02-19T21:53:18.918 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[37667]: cluster 2024-02-19T21:53:17.652615+0000 mon.smithi143 (mon.0) 2 : cluster [INF] mon.smithi143 is new leader, mons smithi143 in quorum (ranks 0) 2024-02-19T21:53:18.918 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[37667]: cluster 2024-02-19T21:53:17.654889+0000 mon.smithi143 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi143=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} 2024-02-19T21:53:18.918 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[37667]: cluster 2024-02-19T21:53:17.657623+0000 mon.smithi143 (mon.0) 4 : cluster [DBG] fsmap 2024-02-19T21:53:18.918 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[37667]: cluster 2024-02-19T21:53:17.660334+0000 mon.smithi143 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T21:53:18.918 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[37667]: cluster 2024-02-19T21:53:17.660717+0000 mon.smithi143 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T21:53:18.918 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[37667]: audit 2024-02-19T21:53:18.374699+0000 mon.smithi143 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.143:0/4177163141' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-19T21:53:19.434 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.413+0000 7f596b126700 1 Processor -- start 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.413+0000 7f596b126700 1 -- start start 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.413+0000 7f596b126700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f5964100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.413+0000 7f596b126700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59640fc8c0 con 0x7f59640703b0 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.413+0000 7f5968ec2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f5964100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f5968ec2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f5964100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41206/0 (socket says 172.21.15.143:41206) 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f5968ec2700 1 -- 172.21.15.143:0/299714485 learned_addr learned my addr 172.21.15.143:0/299714485 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f5968ec2700 1 -- 172.21.15.143:0/299714485 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59641013c0 con 0x7f59640703b0 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f5968ec2700 1 --2- 172.21.15.143:0/299714485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f5964100ed0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f595800b5f0 tx=0x7f5958006ee0).ready entity=mon.0 client_cookie=277ffd92e9c2314f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f59637fe700 1 -- 172.21.15.143:0/299714485 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f595800d600 con 0x7f59640703b0 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f59637fe700 1 -- 172.21.15.143:0/299714485 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595800d760 con 0x7f59640703b0 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f59637fe700 1 -- 172.21.15.143:0/299714485 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f595800da30 con 0x7f59640703b0 2024-02-19T21:53:19.435 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f596b126700 1 -- 172.21.15.143:0/299714485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 msgr2=0x7f5964100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:19.436 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f596b126700 1 --2- 172.21.15.143:0/299714485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f5964100ed0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f595800b5f0 tx=0x7f5958006ee0).stop 2024-02-19T21:53:19.436 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f596b126700 1 -- 172.21.15.143:0/299714485 shutdown_connections 2024-02-19T21:53:19.436 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f596b126700 1 --2- 172.21.15.143:0/299714485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f5964100ed0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:19.436 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.414+0000 7f596b126700 1 -- 172.21.15.143:0/299714485 >> 172.21.15.143:0/299714485 conn(0x7f59640f6530 msgr2=0x7f59640f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:19.436 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.415+0000 7f596b126700 1 -- 172.21.15.143:0/299714485 shutdown_connections 2024-02-19T21:53:19.436 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.415+0000 7f596b126700 1 -- 172.21.15.143:0/299714485 wait complete. 2024-02-19T21:53:19.436 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.415+0000 7f596b126700 1 Processor -- start 2024-02-19T21:53:19.436 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.415+0000 7f596b126700 1 -- start start 2024-02-19T21:53:19.436 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.415+0000 7f596b126700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f596410c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:19.436 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.415+0000 7f596b126700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596410ca40 con 0x7f59640703b0 2024-02-19T21:53:19.436 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.415+0000 7f5968ec2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f596410c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.416+0000 7f5968ec2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f596410c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41210/0 (socket says 172.21.15.143:41210) 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.416+0000 7f5968ec2700 1 -- 172.21.15.143:0/3159617440 learned_addr learned my addr 172.21.15.143:0/3159617440 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.416+0000 7f5968ec2700 1 -- 172.21.15.143:0/3159617440 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595800b200 con 0x7f59640703b0 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.416+0000 7f5968ec2700 1 --2- 172.21.15.143:0/3159617440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f596410c550 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f5958006c70 tx=0x7f5958016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.416+0000 7f59617fa700 1 -- 172.21.15.143:0/3159617440 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5958011b40 con 0x7f59640703b0 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.416+0000 7f59617fa700 1 -- 172.21.15.143:0/3159617440 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5958011ca0 con 0x7f59640703b0 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.416+0000 7f596b126700 1 -- 172.21.15.143:0/3159617440 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596410cc40 con 0x7f59640703b0 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.416+0000 7f59617fa700 1 -- 172.21.15.143:0/3159617440 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f595801d430 con 0x7f59640703b0 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.416+0000 7f596b126700 1 -- 172.21.15.143:0/3159617440 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596410d0d0 con 0x7f59640703b0 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.416+0000 7f59617fa700 1 -- 172.21.15.143:0/3159617440 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f595800f070 con 0x7f59640703b0 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.416+0000 7f59617fa700 1 -- 172.21.15.143:0/3159617440 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5958027460 con 0x7f59640703b0 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.417+0000 7f596b126700 1 -- 172.21.15.143:0/3159617440 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5964045ad0 con 0x7f59640703b0 2024-02-19T21:53:19.437 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.419+0000 7f59617fa700 1 -- 172.21.15.143:0/3159617440 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5958016070 con 0x7f59640703b0 2024-02-19T21:53:19.482 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.483+0000 7f596b126700 1 -- 172.21.15.143:0/3159617440 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f596410e5e0 con 0x7f59640703b0 2024-02-19T21:53:19.488 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.486+0000 7f59617fa700 1 -- 172.21.15.143:0/3159617440 <== mon.0 v2:172.21.15.143:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f595801d5e0 con 0x7f59640703b0 2024-02-19T21:53:19.488 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.486+0000 7f59617fa700 1 -- 172.21.15.143:0/3159617440 <== mon.0 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+491 (secure 0 0 0) 0x7f5958036020 con 0x7f59640703b0 2024-02-19T21:53:19.488 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.488 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: [global] 2024-02-19T21:53:19.488 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: fsid = 38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.143:3300,v1:172.21.15.143:6789] 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: [mgr] 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: [osd] 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-19T21:53:19.489 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-19T21:53:19.490 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-19T21:53:19.490 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-19T21:53:19.490 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-19T21:53:19.490 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.487+0000 7f596b126700 1 -- 172.21.15.143:0/3159617440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 msgr2=0x7f596410c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:19.490 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.487+0000 7f596b126700 1 --2- 172.21.15.143:0/3159617440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f596410c550 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f5958006c70 tx=0x7f5958016000).stop 2024-02-19T21:53:19.490 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.487+0000 7f596b126700 1 -- 172.21.15.143:0/3159617440 shutdown_connections 2024-02-19T21:53:19.490 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.487+0000 7f596b126700 1 --2- 172.21.15.143:0/3159617440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59640703b0 0x7f596410c550 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:19.490 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.487+0000 7f596b126700 1 -- 172.21.15.143:0/3159617440 >> 172.21.15.143:0/3159617440 conn(0x7f59640f6530 msgr2=0x7f59640f8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:19.490 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.487+0000 7f596b126700 1 -- 172.21.15.143:0/3159617440 shutdown_connections 2024-02-19T21:53:19.490 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:19.487+0000 7f596b126700 1 -- 172.21.15.143:0/3159617440 wait complete. 2024-02-19T21:53:19.810 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:19 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[37667]: audit 2024-02-19T21:53:19.484057+0000 mon.smithi143 (mon.0) 8 : audit [INF] from='client.? 172.21.15.143:0/3159617440' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T21:53:19.810 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:19 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[37667]: audit 2024-02-19T21:53:19.487115+0000 mon.smithi143 (mon.0) 9 : audit [INF] from='client.? 172.21.15.143:0/3159617440' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-19T21:53:19.996 INFO:teuthology.orchestra.run.smithi143.stderr:Generating new minimal ceph.conf... 2024-02-19T21:53:19.996 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z -v /var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/mon.smithi143:/var/lib/ceph/mon/ceph-smithi143:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi143/config 2024-02-19T21:53:20.540 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.536+0000 7f054f199700 1 Processor -- start 2024-02-19T21:53:20.540 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.536+0000 7f054f199700 1 -- start start 2024-02-19T21:53:20.541 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.537+0000 7f054f199700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:20.541 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.537+0000 7f054f199700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05480fef80 con 0x7f05480703f0 2024-02-19T21:53:20.541 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.537+0000 7f054cf35700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:20.541 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.537+0000 7f054cf35700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41218/0 (socket says 172.21.15.143:41218) 2024-02-19T21:53:20.541 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.537+0000 7f054cf35700 1 -- 172.21.15.143:0/2107720290 learned_addr learned my addr 172.21.15.143:0/2107720290 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.537+0000 7f054cf35700 1 -- 172.21.15.143:0/2107720290 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05480ff0c0 con 0x7f05480703f0 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.537+0000 7f054cf35700 1 --2- 172.21.15.143:0/2107720290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fea90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f053c00b5f0 tx=0x7f053c006ee0).ready entity=mon.0 client_cookie=734ad7a0a02ae8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.537+0000 7f05477fe700 1 -- 172.21.15.143:0/2107720290 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053c00d600 con 0x7f05480703f0 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.538+0000 7f05477fe700 1 -- 172.21.15.143:0/2107720290 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f053c00dbf0 con 0x7f05480703f0 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.538+0000 7f05477fe700 1 -- 172.21.15.143:0/2107720290 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053c011ce0 con 0x7f05480703f0 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.538+0000 7f054f199700 1 -- 172.21.15.143:0/2107720290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 msgr2=0x7f05480fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.538+0000 7f054f199700 1 --2- 172.21.15.143:0/2107720290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fea90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f053c00b5f0 tx=0x7f053c006ee0).stop 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.538+0000 7f054f199700 1 -- 172.21.15.143:0/2107720290 shutdown_connections 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.538+0000 7f054f199700 1 --2- 172.21.15.143:0/2107720290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fea90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.538+0000 7f054f199700 1 -- 172.21.15.143:0/2107720290 >> 172.21.15.143:0/2107720290 conn(0x7f05480f6530 msgr2=0x7f05480f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.538+0000 7f054f199700 1 -- 172.21.15.143:0/2107720290 shutdown_connections 2024-02-19T21:53:20.542 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.538+0000 7f054f199700 1 -- 172.21.15.143:0/2107720290 wait complete. 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.539+0000 7f054f199700 1 Processor -- start 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.539+0000 7f054f199700 1 -- start start 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.539+0000 7f054f199700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.539+0000 7f054f199700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05480fa990 con 0x7f05480703f0 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.539+0000 7f054cf35700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.539+0000 7f054cf35700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41222/0 (socket says 172.21.15.143:41222) 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.539+0000 7f054cf35700 1 -- 172.21.15.143:0/3196269645 learned_addr learned my addr 172.21.15.143:0/3196269645 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.539+0000 7f054cf35700 1 -- 172.21.15.143:0/3196269645 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c00b200 con 0x7f05480703f0 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.540+0000 7f054cf35700 1 --2- 172.21.15.143:0/3196269645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fc510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f053c011ed0 tx=0x7f053c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.540+0000 7f05457fa700 1 -- 172.21.15.143:0/3196269645 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053c01b570 con 0x7f05480703f0 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.540+0000 7f05457fa700 1 -- 172.21.15.143:0/3196269645 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f053c01bb60 con 0x7f05480703f0 2024-02-19T21:53:20.543 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.540+0000 7f054f199700 1 -- 172.21.15.143:0/3196269645 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05480fab30 con 0x7f05480703f0 2024-02-19T21:53:20.544 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.540+0000 7f05457fa700 1 -- 172.21.15.143:0/3196269645 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053c01aad0 con 0x7f05480703f0 2024-02-19T21:53:20.544 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.540+0000 7f054f199700 1 -- 172.21.15.143:0/3196269645 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05480fafc0 con 0x7f05480703f0 2024-02-19T21:53:20.544 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.540+0000 7f05457fa700 1 -- 172.21.15.143:0/3196269645 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f053c00f070 con 0x7f05480703f0 2024-02-19T21:53:20.544 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.540+0000 7f05457fa700 1 -- 172.21.15.143:0/3196269645 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f053c0258a0 con 0x7f05480703f0 2024-02-19T21:53:20.544 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.541+0000 7f054f199700 1 -- 172.21.15.143:0/3196269645 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0548045ad0 con 0x7f05480703f0 2024-02-19T21:53:20.544 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.543+0000 7f05457fa700 1 -- 172.21.15.143:0/3196269645 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f053c016070 con 0x7f05480703f0 2024-02-19T21:53:20.606 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.607+0000 7f054f199700 1 -- 172.21.15.143:0/3196269645 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f05480008d0 con 0x7f05480703f0 2024-02-19T21:53:20.606 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.607+0000 7f05457fa700 1 -- 172.21.15.143:0/3196269645 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f053c006a70 con 0x7f05480703f0 2024-02-19T21:53:20.608 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.608+0000 7f054f199700 1 -- 172.21.15.143:0/3196269645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 msgr2=0x7f05480fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:20.608 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.608+0000 7f054f199700 1 --2- 172.21.15.143:0/3196269645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fc510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f053c011ed0 tx=0x7f053c006ee0).stop 2024-02-19T21:53:20.608 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.608+0000 7f054f199700 1 -- 172.21.15.143:0/3196269645 shutdown_connections 2024-02-19T21:53:20.608 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.608+0000 7f054f199700 1 --2- 172.21.15.143:0/3196269645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05480703f0 0x7f05480fc510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:20.608 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.608+0000 7f054f199700 1 -- 172.21.15.143:0/3196269645 >> 172.21.15.143:0/3196269645 conn(0x7f05480f6530 msgr2=0x7f05480f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:20.608 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.608+0000 7f054f199700 1 -- 172.21.15.143:0/3196269645 shutdown_connections 2024-02-19T21:53:20.608 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:20.608+0000 7f054f199700 1 -- 172.21.15.143:0/3196269645 wait complete. 2024-02-19T21:53:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[37667]: audit 2024-02-19T21:53:20.608074+0000 mon.smithi143 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.143:0/3196269645' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:53:21.253 INFO:teuthology.orchestra.run.smithi143.stderr:Restarting the monitor... 2024-02-19T21:53:21.253 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl restart ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143 2024-02-19T21:53:21.364 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:21 smithi143 systemd[1]: Stopping Ceph mon.smithi143 for 38187086-cf71-11ee-95bb-87774f69a715... 2024-02-19T21:53:21.645 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[37667]: debug 2024-02-19T21:53:21.364+0000 7f69a8944700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi143 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T21:53:21.645 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[37667]: debug 2024-02-19T21:53:21.364+0000 7f69a8944700 -1 mon.smithi143@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T21:53:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:21 smithi143 podman[37989]: ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143 2024-02-19T21:53:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:21 smithi143 systemd[1]: ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143.service: Succeeded. 2024-02-19T21:53:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:21 smithi143 systemd[1]: Stopped Ceph mon.smithi143 for 38187086-cf71-11ee-95bb-87774f69a715. 2024-02-19T21:53:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:21 smithi143 systemd[1]: Starting Ceph mon.smithi143 for 38187086-cf71-11ee-95bb-87774f69a715... 2024-02-19T21:53:22.460 INFO:teuthology.orchestra.run.smithi143.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-19T21:53:22.460 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-19T21:53:22.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.401+0000 7f7ac28f5700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T21:53:22.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.401+0000 7f7ac28f5700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-19T21:53:22.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.401+0000 7f7ac28f5700 0 pidfile_write: ignore empty --pid-file 2024-02-19T21:53:22.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.405+0000 7f7ac28f5700 0 load: jerasure load: lrc load: isa 2024-02-19T21:53:22.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T21:53:22.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T21:53:22.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Compile date May 13 2021 2024-02-19T21:53:22.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: DB SUMMARY 2024-02-19T21:53:22.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: CURRENT file: CURRENT 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi143/store.db dir, Total Num: 1, files: 000004.sst 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi143/store.db: 000006.log size: 98740 ; 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.create_if_missing: 0 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.env: 0x5636e87651c0 2024-02-19T21:53:22.663 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.fs: Posix File System 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.info_log: 0x5636e9d93960 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.statistics: (nil) 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.use_fsync: 0 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T21:53:22.664 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.db_log_dir: 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi143/store.db 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.write_buffer_manager: 0x5636e9d99fb0 2024-02-19T21:53:22.665 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.unordered_write: 0 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T21:53:22.666 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.row_cache: None 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.wal_filter: None 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T21:53:22.667 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_open_files: -1 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T21:53:22.668 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Compression algorithms supported: 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: kZSTD supported: 0 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi143/store.db/MANIFEST-000005 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.merge_operator: 2024-02-19T21:53:22.669 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_filter: None 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5636e9cabcb0) 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cache_index_and_filter_blocks: 1 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: pin_top_level_index_and_filter: 1 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: index_type: 0 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: data_block_index_type: 0 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: index_shortening: 1 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: data_block_hash_table_util_ratio: 0.750000 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: hash_index_allow_collision: 1 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: checksum: 1 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: no_block_cache: 0 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: block_cache: 0x5636e9ce0f10 2024-02-19T21:53:22.670 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: block_cache_name: BinnedLRUCache 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: block_cache_options: 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: capacity : 536870912 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: num_shard_bits : 4 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: strict_capacity_limit : 0 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: high_pri_pool_ratio: 0.000 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: block_cache_compressed: (nil) 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: persistent_cache: (nil) 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: block_size: 4096 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: block_size_deviation: 10 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: block_restart_interval: 16 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: index_block_restart_interval: 1 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: metadata_block_size: 4096 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: partition_filters: 0 2024-02-19T21:53:22.671 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: use_delta_encoding: 1 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: whole_key_filtering: 1 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: verify_compression: 0 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: read_amp_bytes_per_bit: 0 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: format_version: 2 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: enable_index_compression: 1 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: block_align: 0 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compression: NoCompression 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T21:53:22.672 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.num_levels: 7 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T21:53:22.673 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T21:53:22.674 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T21:53:22.674 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T21:53:22.674 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T21:53:22.674 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T21:53:22.674 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T21:53:22.674 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T21:53:22.674 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T21:53:22.674 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T21:53:22.674 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T21:53:22.674 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T21:53:22.675 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T21:53:22.675 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T21:53:22.675 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T21:53:22.675 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T21:53:22.675 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T21:53:22.675 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T21:53:22.676 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T21:53:22.677 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.ttl: 2592000 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.406+0000 7f7ac28f5700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.407+0000 7f7ac28f5700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi143/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.407+0000 7f7ac28f5700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.407+0000 7f7ac28f5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708379602407913, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.407+0000 7f7ac28f5700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.407+0000 7f7ac28f5700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-19T21:53:22.678 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.408+0000 7f7ac28f5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708379602408762, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94314, "table_properties": {"data_size": 92402, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85433, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708379602, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.408+0000 7f7ac28f5700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.408+0000 7f7ac28f5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708379602409144, "job": 1, "event": "recovery_finished"} 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.409+0000 7f7ac28f5700 4 rocksdb: DB pointer 0x5636e9da5800 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.409+0000 7f7aa970d700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.409+0000 7f7aa970d700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ** DB Stats ** 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T21:53:22.679 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ** Compaction Stats [default] ** 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: L0 2/0 93.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.4 0.00 0.00 1 0.001 0 0 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Sum 2/0 93.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.4 0.00 0.00 1 0.001 0 0 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.4 0.00 0.00 1 0.001 0 0 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ** Compaction Stats [default] ** 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.4 0.00 0.00 1 0.001 0 0 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T21:53:22.680 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Cumulative compaction: 0.00 GB write, 31.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Interval compaction: 0.00 GB write, 31.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ** Compaction Stats [default] ** 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: L0 2/0 93.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.4 0.00 0.00 1 0.001 0 0 2024-02-19T21:53:22.681 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Sum 2/0 93.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.4 0.00 0.00 1 0.001 0 0 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ** Compaction Stats [default] ** 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.4 0.00 0.00 1 0.001 0 0 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Cumulative compaction: 0.00 GB write, 31.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:53:22.682 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.409+0000 7f7ac28f5700 0 starting mon.smithi143 rank 0 at public addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] at bind addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi143 fsid 38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.410+0000 7f7ac28f5700 1 mon.smithi143@-1(???) e1 preinit fsid 38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.410+0000 7f7ac28f5700 0 mon.smithi143@-1(???).mds e1 new map 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.410+0000 7f7ac28f5700 0 mon.smithi143@-1(???).mds e1 print_map 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: e1 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: legacy client fscid: -1 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.683 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: No filesystems configured 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.411+0000 7f7ac28f5700 0 mon.smithi143@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.411+0000 7f7ac28f5700 0 mon.smithi143@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.411+0000 7f7ac28f5700 0 mon.smithi143@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.411+0000 7f7ac28f5700 0 mon.smithi143@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:20.608074+0000 mon.smithi143 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.143:0/3196269645' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.411+0000 7f7ac28f5700 1 mon.smithi143@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta expand map: {default=false} 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta from 'false' to 'false' 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta expanded map: {default=false} 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta expand map: {default=info} 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta from 'info' to 'info' 2024-02-19T21:53:22.684 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta expanded map: {default=info} 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta expand map: {default=daemon} 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta expanded map: {default=daemon} 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta expand map: {default=debug} 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta from 'debug' to 'debug' 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T21:53:22.412+0000 7f7ac28f5700 20 expand_channel_meta expanded map: {default=debug} 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 bash[38060]: 1362eb938b7c3ba0048d5f24c2c3e4c4ea084c15638b66d128e95e26aa4dfe49 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 systemd[1]: Started Ceph mon.smithi143 for 38187086-cf71-11ee-95bb-87774f69a715. 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:22.427380+0000 mon.smithi143 (mon.0) 1 : cluster [INF] mon.smithi143 is new leader, mons smithi143 in quorum (ranks 0) 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:22.427441+0000 mon.smithi143 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi143=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:22.427471+0000 mon.smithi143 (mon.0) 3 : cluster [DBG] fsmap 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:22.427492+0000 mon.smithi143 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T21:53:22.685 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:22.427830+0000 mon.smithi143 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T21:53:22.969 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.968+0000 7f1813331700 1 Processor -- start 2024-02-19T21:53:22.970 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.968+0000 7f1813331700 1 -- start start 2024-02-19T21:53:22.970 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.968+0000 7f1813331700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:22.970 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.968+0000 7f1813331700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0fefc0 con 0x7f180c0703b0 2024-02-19T21:53:22.970 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.969+0000 7f18110cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:22.970 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.969+0000 7f18110cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41226/0 (socket says 172.21.15.143:41226) 2024-02-19T21:53:22.970 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.969+0000 7f18110cd700 1 -- 172.21.15.143:0/1201267523 learned_addr learned my addr 172.21.15.143:0/1201267523 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:22.970 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.969+0000 7f18110cd700 1 -- 172.21.15.143:0/1201267523 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c0ff100 con 0x7f180c0703b0 2024-02-19T21:53:22.970 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.969+0000 7f18110cd700 1 --2- 172.21.15.143:0/1201267523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fead0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f17fc00b5f0 tx=0x7f17fc006ee0).ready entity=mon.0 client_cookie=2cd18577abdd6c50 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:22.970 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.970+0000 7f17fbfff700 1 -- 172.21.15.143:0/1201267523 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17fc00d600 con 0x7f180c0703b0 2024-02-19T21:53:22.970 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.970+0000 7f17fbfff700 1 -- 172.21.15.143:0/1201267523 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f17fc00dbf0 con 0x7f180c0703b0 2024-02-19T21:53:22.970 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.970+0000 7f17fbfff700 1 -- 172.21.15.143:0/1201267523 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17fc011cb0 con 0x7f180c0703b0 2024-02-19T21:53:22.972 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.970+0000 7f1813331700 1 -- 172.21.15.143:0/1201267523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 msgr2=0x7f180c0fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:22.972 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.970+0000 7f1813331700 1 --2- 172.21.15.143:0/1201267523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fead0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f17fc00b5f0 tx=0x7f17fc006ee0).stop 2024-02-19T21:53:22.972 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.970+0000 7f1813331700 1 -- 172.21.15.143:0/1201267523 shutdown_connections 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.970+0000 7f1813331700 1 --2- 172.21.15.143:0/1201267523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fead0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.970+0000 7f1813331700 1 -- 172.21.15.143:0/1201267523 >> 172.21.15.143:0/1201267523 conn(0x7f180c0f6530 msgr2=0x7f180c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.970+0000 7f1813331700 1 -- 172.21.15.143:0/1201267523 shutdown_connections 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.970+0000 7f1813331700 1 -- 172.21.15.143:0/1201267523 wait complete. 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.971+0000 7f1813331700 1 Processor -- start 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.971+0000 7f1813331700 1 -- start start 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.971+0000 7f1813331700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.971+0000 7f1813331700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0fa9d0 con 0x7f180c0703b0 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.971+0000 7f18110cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.971+0000 7f18110cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41234/0 (socket says 172.21.15.143:41234) 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.971+0000 7f18110cd700 1 -- 172.21.15.143:0/531789639 learned_addr learned my addr 172.21.15.143:0/531789639 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.971+0000 7f18110cd700 1 -- 172.21.15.143:0/531789639 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc00b200 con 0x7f180c0703b0 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.971+0000 7f18110cd700 1 --2- 172.21.15.143:0/531789639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fc550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f17fc006800 tx=0x7f17fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.972+0000 7f17f9ffb700 1 -- 172.21.15.143:0/531789639 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17fc01b570 con 0x7f180c0703b0 2024-02-19T21:53:22.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.972+0000 7f17f9ffb700 1 -- 172.21.15.143:0/531789639 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f17fc01bb60 con 0x7f180c0703b0 2024-02-19T21:53:22.974 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.972+0000 7f1813331700 1 -- 172.21.15.143:0/531789639 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f180c0fab70 con 0x7f180c0703b0 2024-02-19T21:53:22.974 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.972+0000 7f17f9ffb700 1 -- 172.21.15.143:0/531789639 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17fc01a9f0 con 0x7f180c0703b0 2024-02-19T21:53:22.974 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.972+0000 7f1813331700 1 -- 172.21.15.143:0/531789639 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180c0fb000 con 0x7f180c0703b0 2024-02-19T21:53:22.974 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.972+0000 7f17f9ffb700 1 -- 172.21.15.143:0/531789639 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f17fc00f070 con 0x7f180c0703b0 2024-02-19T21:53:22.974 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.972+0000 7f17f9ffb700 1 -- 172.21.15.143:0/531789639 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f17fc025d80 con 0x7f180c0703b0 2024-02-19T21:53:22.974 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.973+0000 7f1813331700 1 -- 172.21.15.143:0/531789639 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f180c045ad0 con 0x7f180c0703b0 2024-02-19T21:53:22.974 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:22.975+0000 7f17f9ffb700 1 -- 172.21.15.143:0/531789639 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f17fc016070 con 0x7f180c0703b0 2024-02-19T21:53:23.039 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:23.039+0000 7f1813331700 1 -- 172.21.15.143:0/531789639 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f180c027a00 con 0x7f180c0703b0 2024-02-19T21:53:23.041 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:23.041+0000 7f17f9ffb700 1 -- 172.21.15.143:0/531789639 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f17fc025460 con 0x7f180c0703b0 2024-02-19T21:53:23.042 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:23.042+0000 7f1813331700 1 -- 172.21.15.143:0/531789639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 msgr2=0x7f180c0fc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:23.043 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:23.042+0000 7f1813331700 1 --2- 172.21.15.143:0/531789639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fc550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f17fc006800 tx=0x7f17fc006ee0).stop 2024-02-19T21:53:23.043 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:23.042+0000 7f1813331700 1 -- 172.21.15.143:0/531789639 shutdown_connections 2024-02-19T21:53:23.043 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:23.042+0000 7f1813331700 1 --2- 172.21.15.143:0/531789639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f180c0703b0 0x7f180c0fc550 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:23.043 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:23.042+0000 7f1813331700 1 -- 172.21.15.143:0/531789639 >> 172.21.15.143:0/531789639 conn(0x7f180c0f6530 msgr2=0x7f180c0f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:23.043 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:23.042+0000 7f1813331700 1 -- 172.21.15.143:0/531789639 shutdown_connections 2024-02-19T21:53:23.043 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:23.043+0000 7f1813331700 1 -- 172.21.15.143:0/531789639 wait complete. 2024-02-19T21:53:23.581 INFO:teuthology.orchestra.run.smithi143.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-19T21:53:23.581 INFO:teuthology.orchestra.run.smithi143.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-19T21:53:23.581 INFO:teuthology.orchestra.run.smithi143.stderr:Creating mgr... 2024-02-19T21:53:23.582 INFO:teuthology.orchestra.run.smithi143.stderr:Verifying port 9283 ... 2024-02-19T21:53:23.584 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl daemon-reload 2024-02-19T21:53:23.678 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl stop ceph-38187086-cf71-11ee-95bb-87774f69a715@mgr.smithi143.bissbc 2024-02-19T21:53:23.700 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl reset-failed ceph-38187086-cf71-11ee-95bb-87774f69a715@mgr.smithi143.bissbc 2024-02-19T21:53:23.706 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: Failed to reset failed state of unit ceph-38187086-cf71-11ee-95bb-87774f69a715@mgr.smithi143.bissbc.service: Unit ceph-38187086-cf71-11ee-95bb-87774f69a715@mgr.smithi143.bissbc.service not loaded. 2024-02-19T21:53:23.706 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl enable ceph-38187086-cf71-11ee-95bb-87774f69a715@mgr.smithi143.bissbc 2024-02-19T21:53:23.712 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: Created symlink /etc/systemd/system/ceph-38187086-cf71-11ee-95bb-87774f69a715.target.wants/ceph-38187086-cf71-11ee-95bb-87774f69a715@mgr.smithi143.bissbc.service → /etc/systemd/system/ceph-38187086-cf71-11ee-95bb-87774f69a715@.service. 2024-02-19T21:53:23.793 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl start ceph-38187086-cf71-11ee-95bb-87774f69a715@mgr.smithi143.bissbc 2024-02-19T21:53:24.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:23.042535+0000 mon.smithi143 (mon.0) 6 : audit [INF] from='client.? 172.21.15.143:0/531789639' entity='client.admin' 2024-02-19T21:53:24.436 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T21:53:24.443 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: disabled 2024-02-19T21:53:24.444 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T21:53:24.451 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: inactive 2024-02-19T21:53:24.452 INFO:teuthology.orchestra.run.smithi143.stderr:firewalld.service is not enabled 2024-02-19T21:53:24.452 INFO:teuthology.orchestra.run.smithi143.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T21:53:24.452 INFO:teuthology.orchestra.run.smithi143.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T21:53:24.452 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T21:53:24.459 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: disabled 2024-02-19T21:53:24.460 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T21:53:24.467 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: inactive 2024-02-19T21:53:24.468 INFO:teuthology.orchestra.run.smithi143.stderr:firewalld.service is not enabled 2024-02-19T21:53:24.468 INFO:teuthology.orchestra.run.smithi143.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-19T21:53:24.468 INFO:teuthology.orchestra.run.smithi143.stderr:Waiting for mgr to start... 2024-02-19T21:53:24.468 INFO:teuthology.orchestra.run.smithi143.stderr:Waiting for mgr... 2024-02-19T21:53:24.468 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T21:53:25.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.030+0000 7fd43ee2f700 1 Processor -- start 2024-02-19T21:53:25.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.030+0000 7fd43ee2f700 1 -- start start 2024-02-19T21:53:25.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.030+0000 7fd43ee2f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd438100f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:25.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.030+0000 7fd43ee2f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4380fc980 con 0x7fd4380703b0 2024-02-19T21:53:25.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.031+0000 7fd43cbcb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd438100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:25.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.031+0000 7fd43cbcb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd438100f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41262/0 (socket says 172.21.15.143:41262) 2024-02-19T21:53:25.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.031+0000 7fd43cbcb700 1 -- 172.21.15.143:0/3035379937 learned_addr learned my addr 172.21.15.143:0/3035379937 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:25.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.031+0000 7fd43cbcb700 1 -- 172.21.15.143:0/3035379937 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd438101480 con 0x7fd4380703b0 2024-02-19T21:53:25.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.031+0000 7fd43cbcb700 1 --2- 172.21.15.143:0/3035379937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd438100f90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd42c00b5f0 tx=0x7fd42c006ee0).ready entity=mon.0 client_cookie=fade842161f329a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:25.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.032+0000 7fd4377fe700 1 -- 172.21.15.143:0/3035379937 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd42c00d600 con 0x7fd4380703b0 2024-02-19T21:53:25.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.032+0000 7fd4377fe700 1 -- 172.21.15.143:0/3035379937 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd42c00dbf0 con 0x7fd4380703b0 2024-02-19T21:53:25.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.032+0000 7fd4377fe700 1 -- 172.21.15.143:0/3035379937 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd42c011cb0 con 0x7fd4380703b0 2024-02-19T21:53:25.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.033+0000 7fd43ee2f700 1 -- 172.21.15.143:0/3035379937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 msgr2=0x7fd438100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:25.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.033+0000 7fd43ee2f700 1 --2- 172.21.15.143:0/3035379937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd438100f90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd42c00b5f0 tx=0x7fd42c006ee0).stop 2024-02-19T21:53:25.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.033+0000 7fd43ee2f700 1 -- 172.21.15.143:0/3035379937 shutdown_connections 2024-02-19T21:53:25.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.033+0000 7fd43ee2f700 1 --2- 172.21.15.143:0/3035379937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd438100f90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:25.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.033+0000 7fd43ee2f700 1 -- 172.21.15.143:0/3035379937 >> 172.21.15.143:0/3035379937 conn(0x7fd4380f65b0 msgr2=0x7fd4380f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:25.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.033+0000 7fd43ee2f700 1 -- 172.21.15.143:0/3035379937 shutdown_connections 2024-02-19T21:53:25.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.033+0000 7fd43ee2f700 1 -- 172.21.15.143:0/3035379937 wait complete. 2024-02-19T21:53:25.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.033+0000 7fd43ee2f700 1 Processor -- start 2024-02-19T21:53:25.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.033+0000 7fd43ee2f700 1 -- start start 2024-02-19T21:53:25.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.034+0000 7fd43ee2f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd43810cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:25.035 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.034+0000 7fd43ee2f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43810d2c0 con 0x7fd4380703b0 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.034+0000 7fd43cbcb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd43810cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.034+0000 7fd43cbcb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd43810cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41272/0 (socket says 172.21.15.143:41272) 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.034+0000 7fd43cbcb700 1 -- 172.21.15.143:0/4277760385 learned_addr learned my addr 172.21.15.143:0/4277760385 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.034+0000 7fd43cbcb700 1 -- 172.21.15.143:0/4277760385 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c00b200 con 0x7fd4380703b0 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.034+0000 7fd43cbcb700 1 --2- 172.21.15.143:0/4277760385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd43810cdd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd42c006800 tx=0x7fd42c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.034+0000 7fd4357fa700 1 -- 172.21.15.143:0/4277760385 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd42c01b570 con 0x7fd4380703b0 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.035+0000 7fd43ee2f700 1 -- 172.21.15.143:0/4277760385 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd43810d4c0 con 0x7fd4380703b0 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.035+0000 7fd4357fa700 1 -- 172.21.15.143:0/4277760385 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd42c01bb60 con 0x7fd4380703b0 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.035+0000 7fd4357fa700 1 -- 172.21.15.143:0/4277760385 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd42c01a9f0 con 0x7fd4380703b0 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.035+0000 7fd43ee2f700 1 -- 172.21.15.143:0/4277760385 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43810e100 con 0x7fd4380703b0 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.035+0000 7fd4357fa700 1 -- 172.21.15.143:0/4277760385 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd42c00f070 con 0x7fd4380703b0 2024-02-19T21:53:25.036 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.035+0000 7fd4357fa700 1 -- 172.21.15.143:0/4277760385 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd42c025dd0 con 0x7fd4380703b0 2024-02-19T21:53:25.037 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.036+0000 7fd43ee2f700 1 -- 172.21.15.143:0/4277760385 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43810d950 con 0x7fd4380703b0 2024-02-19T21:53:25.039 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.039+0000 7fd4357fa700 1 -- 172.21.15.143:0/4277760385 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd42c016070 con 0x7fd4380703b0 2024-02-19T21:53:25.113 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.108+0000 7fd43ee2f700 1 -- 172.21.15.143:0/4277760385 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd438102130 con 0x7fd4380703b0 2024-02-19T21:53:25.113 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.109+0000 7fd4357fa700 1 -- 172.21.15.143:0/4277760385 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fd42c025460 con 0x7fd4380703b0 2024-02-19T21:53:25.113 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.113 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: { 2024-02-19T21:53:25.113 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "fsid": "38187086-cf71-11ee-95bb-87774f69a715", 2024-02-19T21:53:25.113 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "health": { 2024-02-19T21:53:25.113 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T21:53:25.113 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T21:53:25.113 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 0 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: ], 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "smithi143" 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: ], 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T21:53:25.114 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T21:53:25.115 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T21:53:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T21:53:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T21:53:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T21:53:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T21:53:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T21:53:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T21:53:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "available": false, 2024-02-19T21:53:25.116 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T21:53:25.118 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T21:53:25.118 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "iostat", 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "restful" 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: ], 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "modified": "2024-02-19T21:53:17.656142+0000", 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: } 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.110+0000 7fd43ee2f700 1 -- 172.21.15.143:0/4277760385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 msgr2=0x7fd43810cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.110+0000 7fd43ee2f700 1 --2- 172.21.15.143:0/4277760385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd43810cdd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd42c006800 tx=0x7fd42c006ee0).stop 2024-02-19T21:53:25.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.110+0000 7fd43ee2f700 1 -- 172.21.15.143:0/4277760385 shutdown_connections 2024-02-19T21:53:25.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.110+0000 7fd43ee2f700 1 --2- 172.21.15.143:0/4277760385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4380703b0 0x7fd43810cdd0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:25.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.110+0000 7fd43ee2f700 1 -- 172.21.15.143:0/4277760385 >> 172.21.15.143:0/4277760385 conn(0x7fd4380f65b0 msgr2=0x7fd4380f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:25.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.110+0000 7fd43ee2f700 1 -- 172.21.15.143:0/4277760385 shutdown_connections 2024-02-19T21:53:25.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:25.110+0000 7fd43ee2f700 1 -- 172.21.15.143:0/4277760385 wait complete. 2024-02-19T21:53:25.404 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:25 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:25.109714+0000 mon.smithi143 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.143:0/4277760385' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T21:53:25.616 INFO:teuthology.orchestra.run.smithi143.stderr:mgr not available, waiting (1/15)... 2024-02-19T21:53:27.618 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T21:53:28.178 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.174+0000 7fbb24e44700 1 Processor -- start 2024-02-19T21:53:28.178 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.174+0000 7fbb24e44700 1 -- start start 2024-02-19T21:53:28.178 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.174+0000 7fbb24e44700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb200fc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:28.178 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.174+0000 7fbb24e44700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb200fcc60 con 0x7fbb200fc350 2024-02-19T21:53:28.178 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.175+0000 7fbb1e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb200fc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:28.178 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.175+0000 7fbb1e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb200fc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41288/0 (socket says 172.21.15.143:41288) 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.175+0000 7fbb1e59c700 1 -- 172.21.15.143:0/973531042 learned_addr learned my addr 172.21.15.143:0/973531042 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.175+0000 7fbb1e59c700 1 -- 172.21.15.143:0/973531042 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb200fcda0 con 0x7fbb200fc350 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.175+0000 7fbb1e59c700 1 --2- 172.21.15.143:0/973531042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb200fc770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fbb1000b5f0 tx=0x7fbb10006ee0).ready entity=mon.0 client_cookie=8ce7c65b5583086 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.175+0000 7fbb1d59a700 1 -- 172.21.15.143:0/973531042 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1000d600 con 0x7fbb200fc350 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.175+0000 7fbb1d59a700 1 -- 172.21.15.143:0/973531042 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbb1000dbf0 con 0x7fbb200fc350 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.175+0000 7fbb1d59a700 1 -- 172.21.15.143:0/973531042 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb10011cb0 con 0x7fbb200fc350 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.176+0000 7fbb24e44700 1 -- 172.21.15.143:0/973531042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 msgr2=0x7fbb200fc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.176+0000 7fbb24e44700 1 --2- 172.21.15.143:0/973531042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb200fc770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fbb1000b5f0 tx=0x7fbb10006ee0).stop 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.176+0000 7fbb24e44700 1 -- 172.21.15.143:0/973531042 shutdown_connections 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.176+0000 7fbb24e44700 1 --2- 172.21.15.143:0/973531042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb200fc770 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.176+0000 7fbb24e44700 1 -- 172.21.15.143:0/973531042 >> 172.21.15.143:0/973531042 conn(0x7fbb200f5d20 msgr2=0x7fbb200f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:28.179 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.176+0000 7fbb24e44700 1 -- 172.21.15.143:0/973531042 shutdown_connections 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.176+0000 7fbb24e44700 1 -- 172.21.15.143:0/973531042 wait complete. 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.176+0000 7fbb24e44700 1 Processor -- start 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.176+0000 7fbb24e44700 1 -- start start 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb24e44700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb2010ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb24e44700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2010b080 con 0x7fbb200fc350 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb1e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb2010ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb1e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb2010ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41298/0 (socket says 172.21.15.143:41298) 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb1e59c700 1 -- 172.21.15.143:0/2840238610 learned_addr learned my addr 172.21.15.143:0/2840238610 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb1e59c700 1 -- 172.21.15.143:0/2840238610 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1000b200 con 0x7fbb200fc350 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb1e59c700 1 --2- 172.21.15.143:0/2840238610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb2010ab90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbb10006800 tx=0x7fbb10006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb0effd700 1 -- 172.21.15.143:0/2840238610 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1001b570 con 0x7fbb200fc350 2024-02-19T21:53:28.180 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb0effd700 1 -- 172.21.15.143:0/2840238610 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbb1001bb60 con 0x7fbb200fc350 2024-02-19T21:53:28.181 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb24e44700 1 -- 172.21.15.143:0/2840238610 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2010b280 con 0x7fbb200fc350 2024-02-19T21:53:28.181 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb0effd700 1 -- 172.21.15.143:0/2840238610 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb1001a9f0 con 0x7fbb200fc350 2024-02-19T21:53:28.181 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.177+0000 7fbb24e44700 1 -- 172.21.15.143:0/2840238610 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2010b710 con 0x7fbb200fc350 2024-02-19T21:53:28.181 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.178+0000 7fbb0effd700 1 -- 172.21.15.143:0/2840238610 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fbb1000f070 con 0x7fbb200fc350 2024-02-19T21:53:28.181 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.178+0000 7fbb0effd700 1 -- 172.21.15.143:0/2840238610 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbb10025d00 con 0x7fbb200fc350 2024-02-19T21:53:28.181 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.178+0000 7fbb24e44700 1 -- 172.21.15.143:0/2840238610 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb20045ad0 con 0x7fbb200fc350 2024-02-19T21:53:28.181 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.180+0000 7fbb0effd700 1 -- 172.21.15.143:0/2840238610 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fbb10016070 con 0x7fbb200fc350 2024-02-19T21:53:28.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.248+0000 7fbb24e44700 1 -- 172.21.15.143:0/2840238610 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbb200fbc60 con 0x7fbb200fc350 2024-02-19T21:53:28.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.249+0000 7fbb0effd700 1 -- 172.21.15.143:0/2840238610 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fbb10025460 con 0x7fbb200fc350 2024-02-19T21:53:28.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: { 2024-02-19T21:53:28.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "fsid": "38187086-cf71-11ee-95bb-87774f69a715", 2024-02-19T21:53:28.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "health": { 2024-02-19T21:53:28.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T21:53:28.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T21:53:28.254 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T21:53:28.254 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:28.254 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T21:53:28.255 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T21:53:28.255 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 0 2024-02-19T21:53:28.255 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: ], 2024-02-19T21:53:28.255 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T21:53:28.255 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "smithi143" 2024-02-19T21:53:28.255 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: ], 2024-02-19T21:53:28.255 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-19T21:53:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T21:53:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T21:53:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T21:53:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T21:53:28.256 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T21:53:28.257 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T21:53:28.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T21:53:28.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T21:53:28.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:28.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T21:53:28.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:28.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T21:53:28.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T21:53:28.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:28.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T21:53:28.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "available": false, 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "iostat", 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "restful" 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: ], 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "modified": "2024-02-19T21:53:17.656142+0000", 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: } 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.250+0000 7fbb24e44700 1 -- 172.21.15.143:0/2840238610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 msgr2=0x7fbb2010ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.250+0000 7fbb24e44700 1 --2- 172.21.15.143:0/2840238610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb2010ab90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbb10006800 tx=0x7fbb10006ee0).stop 2024-02-19T21:53:28.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.250+0000 7fbb24e44700 1 -- 172.21.15.143:0/2840238610 shutdown_connections 2024-02-19T21:53:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.250+0000 7fbb24e44700 1 --2- 172.21.15.143:0/2840238610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb200fc350 0x7fbb2010ab90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.250+0000 7fbb24e44700 1 -- 172.21.15.143:0/2840238610 >> 172.21.15.143:0/2840238610 conn(0x7fbb200f5d20 msgr2=0x7fbb200fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.250+0000 7fbb24e44700 1 -- 172.21.15.143:0/2840238610 shutdown_connections 2024-02-19T21:53:28.260 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:28.250+0000 7fbb24e44700 1 -- 172.21.15.143:0/2840238610 wait complete. 2024-02-19T21:53:28.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:28.249879+0000 mon.smithi143 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.143:0/2840238610' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T21:53:28.929 INFO:teuthology.orchestra.run.smithi143.stderr:mgr not available, waiting (2/15)... 2024-02-19T21:53:29.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:29.266639+0000 mon.smithi143 (mon.0) 9 : cluster [INF] Activating manager daemon smithi143.bissbc 2024-02-19T21:53:29.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:29.269322+0000 mon.smithi143 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi143.bissbc(active, starting, since 0.00277583s) 2024-02-19T21:53:29.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:29.270179+0000 mon.smithi143 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.143:0/2234500257' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T21:53:29.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:29.270303+0000 mon.smithi143 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.143:0/2234500257' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T21:53:29.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:29.270481+0000 mon.smithi143 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.143:0/2234500257' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T21:53:29.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:29.270661+0000 mon.smithi143 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.143:0/2234500257' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-02-19T21:53:29.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:29.270972+0000 mon.smithi143 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.143:0/2234500257' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mgr metadata", "who": "smithi143.bissbc", "id": "smithi143.bissbc"}]: dispatch 2024-02-19T21:53:29.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:29.272716+0000 mon.smithi143 (mon.0) 16 : cluster [INF] Manager daemon smithi143.bissbc is now available 2024-02-19T21:53:29.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:29.280014+0000 mon.smithi143 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.143:0/2234500257' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:29.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:29.284404+0000 mon.smithi143 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.143:0/2234500257' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:53:29.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:29.285503+0000 mon.smithi143 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.143:0/2234500257' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T21:53:29.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:29.290351+0000 mon.smithi143 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.143:0/2234500257' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:29.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:29.292009+0000 mon.smithi143 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.143:0/2234500257' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:30.932 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T21:53:31.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:31 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:30.271920+0000 mon.smithi143 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi143.bissbc(active, since 1.00537s) 2024-02-19T21:53:33.030 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.026+0000 7f8dd67b2700 1 Processor -- start 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.026+0000 7f8dd67b2700 1 -- start start 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.027+0000 7f8dd67b2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd00feb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.027+0000 7f8dd67b2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00ff040 con 0x7f8dd00703f0 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.027+0000 7f8dcffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd00feb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.027+0000 7f8dcffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd00feb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:47034/0 (socket says 172.21.15.143:47034) 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.027+0000 7f8dcffff700 1 -- 172.21.15.143:0/803057135 learned_addr learned my addr 172.21.15.143:0/803057135 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.027+0000 7f8dcffff700 1 -- 172.21.15.143:0/803057135 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd00ff180 con 0x7f8dd00703f0 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.027+0000 7f8dcffff700 1 --2- 172.21.15.143:0/803057135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd00feb50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8dc000b5f0 tx=0x7f8dc0006ee0).ready entity=mon.0 client_cookie=65ab08d168f52ff1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.027+0000 7f8dceffd700 1 -- 172.21.15.143:0/803057135 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc000d600 con 0x7f8dd00703f0 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.028+0000 7f8dceffd700 1 -- 172.21.15.143:0/803057135 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8dc000dbf0 con 0x7f8dd00703f0 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.028+0000 7f8dceffd700 1 -- 172.21.15.143:0/803057135 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc0011ce0 con 0x7f8dd00703f0 2024-02-19T21:53:33.031 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.028+0000 7f8dd67b2700 1 -- 172.21.15.143:0/803057135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 msgr2=0x7f8dd00feb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.028+0000 7f8dd67b2700 1 --2- 172.21.15.143:0/803057135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd00feb50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8dc000b5f0 tx=0x7f8dc0006ee0).stop 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.028+0000 7f8dd67b2700 1 -- 172.21.15.143:0/803057135 shutdown_connections 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.028+0000 7f8dd67b2700 1 --2- 172.21.15.143:0/803057135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd00feb50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.028+0000 7f8dd67b2700 1 -- 172.21.15.143:0/803057135 >> 172.21.15.143:0/803057135 conn(0x7f8dd00f65b0 msgr2=0x7f8dd00f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.028+0000 7f8dd67b2700 1 -- 172.21.15.143:0/803057135 shutdown_connections 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.028+0000 7f8dd67b2700 1 -- 172.21.15.143:0/803057135 wait complete. 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.029+0000 7f8dd67b2700 1 Processor -- start 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.029+0000 7f8dd67b2700 1 -- start start 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.029+0000 7f8dd67b2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd010abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.029+0000 7f8dd67b2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd010b0d0 con 0x7f8dd00703f0 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.029+0000 7f8dcffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd010abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.029+0000 7f8dcffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd010abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:47046/0 (socket says 172.21.15.143:47046) 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.029+0000 7f8dcffff700 1 -- 172.21.15.143:0/1024542925 learned_addr learned my addr 172.21.15.143:0/1024542925 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.030+0000 7f8dcffff700 1 -- 172.21.15.143:0/1024542925 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc000b200 con 0x7f8dd00703f0 2024-02-19T21:53:33.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.030+0000 7f8dcffff700 1 --2- 172.21.15.143:0/1024542925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd010abe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8dc000df90 tx=0x7f8dc0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:33.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.030+0000 7f8dccff9700 1 -- 172.21.15.143:0/1024542925 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc001b570 con 0x7f8dd00703f0 2024-02-19T21:53:33.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.030+0000 7f8dd67b2700 1 -- 172.21.15.143:0/1024542925 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd010b2d0 con 0x7f8dd00703f0 2024-02-19T21:53:33.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.030+0000 7f8dccff9700 1 -- 172.21.15.143:0/1024542925 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8dc001bb60 con 0x7f8dd00703f0 2024-02-19T21:53:33.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.030+0000 7f8dccff9700 1 -- 172.21.15.143:0/1024542925 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dc001ab80 con 0x7f8dd00703f0 2024-02-19T21:53:33.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.030+0000 7f8dd67b2700 1 -- 172.21.15.143:0/1024542925 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd010b760 con 0x7f8dd00703f0 2024-02-19T21:53:33.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.031+0000 7f8dccff9700 1 -- 172.21.15.143:0/1024542925 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f8dc000f070 con 0x7f8dd00703f0 2024-02-19T21:53:33.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.031+0000 7f8dccff9700 1 --2- 172.21.15.143:0/1024542925 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f8dbc02fdc0 0x7f8dbc032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:33.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.031+0000 7f8dd67b2700 1 -- 172.21.15.143:0/1024542925 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd0045ad0 con 0x7f8dd00703f0 2024-02-19T21:53:33.033 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.031+0000 7f8dccff9700 1 -- 172.21.15.143:0/1024542925 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8dc0040ff0 con 0x7f8dd00703f0 2024-02-19T21:53:33.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.031+0000 7f8dcf7fe700 1 --2- 172.21.15.143:0/1024542925 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f8dbc02fdc0 0x7f8dbc032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:33.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.031+0000 7f8dcf7fe700 1 --2- 172.21.15.143:0/1024542925 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f8dbc02fdc0 0x7f8dbc032280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f8dc40096a0 tx=0x7f8dc4006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:33.035 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.036+0000 7f8dccff9700 1 -- 172.21.15.143:0/1024542925 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8dc0010be0 con 0x7f8dd00703f0 2024-02-19T21:53:33.239 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.238+0000 7f8dd67b2700 1 -- 172.21.15.143:0/1024542925 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8dd00ffe50 con 0x7f8dd00703f0 2024-02-19T21:53:33.239 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.239+0000 7f8dccff9700 1 -- 172.21.15.143:0/1024542925 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f8dc00107d0 con 0x7f8dd00703f0 2024-02-19T21:53:33.239 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.239 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: { 2024-02-19T21:53:33.239 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "fsid": "38187086-cf71-11ee-95bb-87774f69a715", 2024-02-19T21:53:33.240 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "health": { 2024-02-19T21:53:33.240 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T21:53:33.240 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T21:53:33.240 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T21:53:33.240 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:33.240 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T21:53:33.240 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T21:53:33.240 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 0 2024-02-19T21:53:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: ], 2024-02-19T21:53:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T21:53:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "smithi143" 2024-02-19T21:53:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: ], 2024-02-19T21:53:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-02-19T21:53:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T21:53:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:33.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T21:53:33.250 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "available": true, 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T21:53:33.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "iostat", 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "restful" 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: ], 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "modified": "2024-02-19T21:53:17.656142+0000", 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: }, 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: } 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.241+0000 7f8dd67b2700 1 -- 172.21.15.143:0/1024542925 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f8dbc02fdc0 msgr2=0x7f8dbc032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:33.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.241+0000 7f8dd67b2700 1 --2- 172.21.15.143:0/1024542925 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f8dbc02fdc0 0x7f8dbc032280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f8dc40096a0 tx=0x7f8dc4006b40).stop 2024-02-19T21:53:33.253 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.241+0000 7f8dd67b2700 1 -- 172.21.15.143:0/1024542925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 msgr2=0x7f8dd010abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:33.253 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.241+0000 7f8dd67b2700 1 --2- 172.21.15.143:0/1024542925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd010abe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8dc000df90 tx=0x7f8dc0006ee0).stop 2024-02-19T21:53:33.253 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.241+0000 7f8dd67b2700 1 -- 172.21.15.143:0/1024542925 shutdown_connections 2024-02-19T21:53:33.253 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.241+0000 7f8dd67b2700 1 --2- 172.21.15.143:0/1024542925 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f8dbc02fdc0 0x7f8dbc032280 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:33.253 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.241+0000 7f8dd67b2700 1 --2- 172.21.15.143:0/1024542925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dd00703f0 0x7f8dd010abe0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:33.253 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.241+0000 7f8dd67b2700 1 -- 172.21.15.143:0/1024542925 >> 172.21.15.143:0/1024542925 conn(0x7f8dd00f65b0 msgr2=0x7f8dd00f7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:33.253 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.241+0000 7f8dd67b2700 1 -- 172.21.15.143:0/1024542925 shutdown_connections 2024-02-19T21:53:33.253 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:33.241+0000 7f8dd67b2700 1 -- 172.21.15.143:0/1024542925 wait complete. 2024-02-19T21:53:33.519 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:33 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:32.273163+0000 mon.smithi143 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi143.bissbc(active, since 3s) 2024-02-19T21:53:33.519 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:33 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:33.239696+0000 mon.smithi143 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.143:0/1024542925' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T21:53:33.764 INFO:teuthology.orchestra.run.smithi143.stderr:mgr is available 2024-02-19T21:53:33.764 INFO:teuthology.orchestra.run.smithi143.stderr:Enabling cephadm module... 2024-02-19T21:53:33.764 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-19T21:53:34.316 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.313+0000 7f2cfa4d1700 1 Processor -- start 2024-02-19T21:53:34.316 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.313+0000 7f2cfa4d1700 1 -- start start 2024-02-19T21:53:34.316 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.313+0000 7f2cfa4d1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.313+0000 7f2cfa4d1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf40fefa0 con 0x7f2cf40703d0 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.313+0000 7f2cf3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.313+0000 7f2cf3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:47052/0 (socket says 172.21.15.143:47052) 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.313+0000 7f2cf3fff700 1 -- 172.21.15.143:0/1497078782 learned_addr learned my addr 172.21.15.143:0/1497078782 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.314+0000 7f2cf3fff700 1 -- 172.21.15.143:0/1497078782 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf40ff0e0 con 0x7f2cf40703d0 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.314+0000 7f2cf3fff700 1 --2- 172.21.15.143:0/1497078782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40feab0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f2ce800b5f0 tx=0x7f2ce8006ee0).ready entity=mon.0 client_cookie=d3d96be4ff12c8fd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.314+0000 7f2cf2ffd700 1 -- 172.21.15.143:0/1497078782 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ce800d600 con 0x7f2cf40703d0 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.314+0000 7f2cf2ffd700 1 -- 172.21.15.143:0/1497078782 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2ce800dbf0 con 0x7f2cf40703d0 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.314+0000 7f2cf2ffd700 1 -- 172.21.15.143:0/1497078782 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ce8011cb0 con 0x7f2cf40703d0 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.314+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1497078782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 msgr2=0x7f2cf40feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.315+0000 7f2cfa4d1700 1 --2- 172.21.15.143:0/1497078782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40feab0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f2ce800b5f0 tx=0x7f2ce8006ee0).stop 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.315+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1497078782 shutdown_connections 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.315+0000 7f2cfa4d1700 1 --2- 172.21.15.143:0/1497078782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40feab0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:34.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.315+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1497078782 >> 172.21.15.143:0/1497078782 conn(0x7f2cf40f6530 msgr2=0x7f2cf40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.315+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1497078782 shutdown_connections 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.315+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1497078782 wait complete. 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.315+0000 7f2cfa4d1700 1 Processor -- start 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.315+0000 7f2cfa4d1700 1 -- start start 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.315+0000 7f2cfa4d1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40fc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.315+0000 7f2cfa4d1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf40fa9b0 con 0x7f2cf40703d0 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.316+0000 7f2cf3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.316+0000 7f2cf3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40fc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:47062/0 (socket says 172.21.15.143:47062) 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.316+0000 7f2cf3fff700 1 -- 172.21.15.143:0/1816735519 learned_addr learned my addr 172.21.15.143:0/1816735519 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.316+0000 7f2cf3fff700 1 -- 172.21.15.143:0/1816735519 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce800b200 con 0x7f2cf40703d0 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.316+0000 7f2cf3fff700 1 --2- 172.21.15.143:0/1816735519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40fc530 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f2ce8006a20 tx=0x7f2ce8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.316+0000 7f2cf0ff9700 1 -- 172.21.15.143:0/1816735519 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ce801b570 con 0x7f2cf40703d0 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.316+0000 7f2cf0ff9700 1 -- 172.21.15.143:0/1816735519 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2ce801bb60 con 0x7f2cf40703d0 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.316+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1816735519 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf40fab50 con 0x7f2cf40703d0 2024-02-19T21:53:34.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.316+0000 7f2cf0ff9700 1 -- 172.21.15.143:0/1816735519 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ce801aa20 con 0x7f2cf40703d0 2024-02-19T21:53:34.319 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.316+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1816735519 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf40fafe0 con 0x7f2cf40703d0 2024-02-19T21:53:34.319 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.317+0000 7f2cf0ff9700 1 -- 172.21.15.143:0/1816735519 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f2ce800f070 con 0x7f2cf40703d0 2024-02-19T21:53:34.319 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.317+0000 7f2cf0ff9700 1 --2- 172.21.15.143:0/1816735519 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f2ce002fe10 0x7f2ce00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:34.319 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.317+0000 7f2cf0ff9700 1 -- 172.21.15.143:0/1816735519 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2ce80420c0 con 0x7f2cf40703d0 2024-02-19T21:53:34.319 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.317+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1816735519 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf4045ad0 con 0x7f2cf40703d0 2024-02-19T21:53:34.320 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.317+0000 7f2cf37fe700 1 --2- 172.21.15.143:0/1816735519 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f2ce002fe10 0x7f2ce00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:34.320 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.318+0000 7f2cf37fe700 1 --2- 172.21.15.143:0/1816735519 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f2ce002fe10 0x7f2ce00322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2ce40096a0 tx=0x7f2ce4006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:34.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.322+0000 7f2cf0ff9700 1 -- 172.21.15.143:0/1816735519 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ce8016070 con 0x7f2cf40703d0 2024-02-19T21:53:34.509 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.509+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1816735519 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f2cf4027a00 con 0x7f2cf40703d0 2024-02-19T21:53:34.561 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.561+0000 7f2cf0ff9700 1 -- 172.21.15.143:0/1816735519 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f2ce801bcd0 con 0x7f2cf40703d0 2024-02-19T21:53:34.562 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.563+0000 7f2cf0ff9700 1 -- 172.21.15.143:0/1816735519 <== mon.0 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f2ce8023be0 con 0x7f2cf40703d0 2024-02-19T21:53:34.565 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.564+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1816735519 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f2ce002fe10 msgr2=0x7f2ce00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:34.565 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.564+0000 7f2cfa4d1700 1 --2- 172.21.15.143:0/1816735519 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f2ce002fe10 0x7f2ce00322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2ce40096a0 tx=0x7f2ce4006b40).stop 2024-02-19T21:53:34.565 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.564+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1816735519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 msgr2=0x7f2cf40fc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:34.565 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.564+0000 7f2cfa4d1700 1 --2- 172.21.15.143:0/1816735519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40fc530 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f2ce8006a20 tx=0x7f2ce8006ee0).stop 2024-02-19T21:53:34.565 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.564+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1816735519 shutdown_connections 2024-02-19T21:53:34.565 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.564+0000 7f2cfa4d1700 1 --2- 172.21.15.143:0/1816735519 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f2ce002fe10 0x7f2ce00322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:34.565 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.564+0000 7f2cfa4d1700 1 --2- 172.21.15.143:0/1816735519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40703d0 0x7f2cf40fc530 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:34.566 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.565+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1816735519 >> 172.21.15.143:0/1816735519 conn(0x7f2cf40f6530 msgr2=0x7f2cf40f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:34.566 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.565+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1816735519 shutdown_connections 2024-02-19T21:53:34.566 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:34.565+0000 7f2cfa4d1700 1 -- 172.21.15.143:0/1816735519 wait complete. 2024-02-19T21:53:34.908 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:34.510341+0000 mon.smithi143 (mon.0) 25 : audit [INF] from='client.? 172.21.15.143:0/1816735519' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-19T21:53:35.119 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T21:53:35.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:34.563676+0000 mon.smithi143 (mon.0) 26 : audit [INF] from='client.? 172.21.15.143:0/1816735519' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-19T21:53:35.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:34.563757+0000 mon.smithi143 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi143.bissbc(active, since 5s) 2024-02-19T21:53:35.757 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.755+0000 7fd9a73a9700 1 Processor -- start 2024-02-19T21:53:35.757 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.755+0000 7fd9a73a9700 1 -- start start 2024-02-19T21:53:35.757 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.755+0000 7fd9a73a9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a00fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:35.757 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.756+0000 7fd9a73a9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a00fef40 con 0x7fd9a0070430 2024-02-19T21:53:35.757 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.756+0000 7fd9a5145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a00fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.756+0000 7fd9a5145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a00fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:47086/0 (socket says 172.21.15.143:47086) 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.756+0000 7fd9a5145700 1 -- 172.21.15.143:0/1680848735 learned_addr learned my addr 172.21.15.143:0/1680848735 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.756+0000 7fd9a5145700 1 -- 172.21.15.143:0/1680848735 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a00ff080 con 0x7fd9a0070430 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.756+0000 7fd9a5145700 1 --2- 172.21.15.143:0/1680848735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a00fea50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fd99c00b5f0 tx=0x7fd99c006ee0).ready entity=mon.0 client_cookie=1536f7df42bb5c85 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.756+0000 7fd98ffff700 1 -- 172.21.15.143:0/1680848735 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd99c00d600 con 0x7fd9a0070430 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.756+0000 7fd98ffff700 1 -- 172.21.15.143:0/1680848735 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd99c00dbf0 con 0x7fd9a0070430 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.756+0000 7fd98ffff700 1 -- 172.21.15.143:0/1680848735 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd99c011cb0 con 0x7fd9a0070430 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.757+0000 7fd9a73a9700 1 -- 172.21.15.143:0/1680848735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 msgr2=0x7fd9a00fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.757+0000 7fd9a73a9700 1 --2- 172.21.15.143:0/1680848735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a00fea50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fd99c00b5f0 tx=0x7fd99c006ee0).stop 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.757+0000 7fd9a73a9700 1 -- 172.21.15.143:0/1680848735 shutdown_connections 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.757+0000 7fd9a73a9700 1 --2- 172.21.15.143:0/1680848735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a00fea50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.757+0000 7fd9a73a9700 1 -- 172.21.15.143:0/1680848735 >> 172.21.15.143:0/1680848735 conn(0x7fd9a00f6530 msgr2=0x7fd9a00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.757+0000 7fd9a73a9700 1 -- 172.21.15.143:0/1680848735 shutdown_connections 2024-02-19T21:53:35.758 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.757+0000 7fd9a73a9700 1 -- 172.21.15.143:0/1680848735 wait complete. 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.758+0000 7fd9a73a9700 1 Processor -- start 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.758+0000 7fd9a73a9700 1 -- start start 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.758+0000 7fd9a73a9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a010ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.758+0000 7fd9a73a9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a010b080 con 0x7fd9a0070430 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.758+0000 7fd9a5145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a010ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.758+0000 7fd9a5145700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a010ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:47096/0 (socket says 172.21.15.143:47096) 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.758+0000 7fd9a5145700 1 -- 172.21.15.143:0/3086271352 learned_addr learned my addr 172.21.15.143:0/3086271352 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.759+0000 7fd9a5145700 1 -- 172.21.15.143:0/3086271352 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c00b200 con 0x7fd9a0070430 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.759+0000 7fd9a5145700 1 --2- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a010ab90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fd99c00df90 tx=0x7fd99c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.759+0000 7fd98dffb700 1 -- 172.21.15.143:0/3086271352 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd99c01b410 con 0x7fd9a0070430 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.759+0000 7fd9a73a9700 1 -- 172.21.15.143:0/3086271352 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a010b280 con 0x7fd9a0070430 2024-02-19T21:53:35.761 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.759+0000 7fd98dffb700 1 -- 172.21.15.143:0/3086271352 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd99c01ba00 con 0x7fd9a0070430 2024-02-19T21:53:35.762 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.759+0000 7fd98dffb700 1 -- 172.21.15.143:0/3086271352 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd99c01aa30 con 0x7fd9a0070430 2024-02-19T21:53:35.762 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.759+0000 7fd9a73a9700 1 -- 172.21.15.143:0/3086271352 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a010b710 con 0x7fd9a0070430 2024-02-19T21:53:35.762 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.760+0000 7fd98dffb700 1 -- 172.21.15.143:0/3086271352 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd99c00f070 con 0x7fd9a0070430 2024-02-19T21:53:35.762 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.760+0000 7fd98dffb700 1 --2- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7fd98802fe10 0x7fd9880322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:35.762 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.760+0000 7fd98dffb700 1 -- 172.21.15.143:0/3086271352 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd99c0410c0 con 0x7fd9a0070430 2024-02-19T21:53:35.762 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.760+0000 7fd9a4944700 1 -- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7fd98802fe10 msgr2=0x7fd9880322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/4035514323 2024-02-19T21:53:35.762 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.760+0000 7fd9a4944700 1 --2- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7fd98802fe10 0x7fd9880322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T21:53:35.762 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.760+0000 7fd9a73a9700 1 -- 172.21.15.143:0/3086271352 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a0045ad0 con 0x7fd9a0070430 2024-02-19T21:53:35.764 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.765+0000 7fd98dffb700 1 -- 172.21.15.143:0/3086271352 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd99c016070 con 0x7fd9a0070430 2024-02-19T21:53:35.961 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.961+0000 7fd9a4944700 1 -- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7fd98802fe10 msgr2=0x7fd9880322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/4035514323 2024-02-19T21:53:35.961 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.961+0000 7fd9a4944700 1 --2- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7fd98802fe10 0x7fd9880322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T21:53:35.993 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.992+0000 7fd9a73a9700 1 -- 172.21.15.143:0/3086271352 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd9a00008d0 con 0x7fd9a0070430 2024-02-19T21:53:35.993 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.993+0000 7fd98dffb700 1 -- 172.21.15.143:0/3086271352 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fd99c022d20 con 0x7fd9a0070430 2024-02-19T21:53:35.993 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: { 2024-02-19T21:53:35.993 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-19T21:53:35.993 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "available": true, 2024-02-19T21:53:35.993 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "active_name": "smithi143.bissbc", 2024-02-19T21:53:35.993 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T21:53:35.993 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: } 2024-02-19T21:53:35.995 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.995+0000 7fd9a73a9700 1 -- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7fd98802fe10 msgr2=0x7fd9880322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:53:35.995 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.995+0000 7fd9a73a9700 1 --2- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7fd98802fe10 0x7fd9880322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:53:35.995 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.995+0000 7fd9a73a9700 1 -- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 msgr2=0x7fd9a010ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:35.995 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.995+0000 7fd9a73a9700 1 --2- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a010ab90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fd99c00df90 tx=0x7fd99c006ee0).stop 2024-02-19T21:53:35.995 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.995+0000 7fd9a73a9700 1 -- 172.21.15.143:0/3086271352 shutdown_connections 2024-02-19T21:53:35.995 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.995+0000 7fd9a73a9700 1 --2- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7fd98802fe10 0x7fd9880322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:53:35.996 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.995+0000 7fd9a73a9700 1 --2- 172.21.15.143:0/3086271352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0070430 0x7fd9a010ab90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:35.996 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.995+0000 7fd9a73a9700 1 -- 172.21.15.143:0/3086271352 >> 172.21.15.143:0/3086271352 conn(0x7fd9a00f6530 msgr2=0x7fd9a00f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:35.996 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.995+0000 7fd9a73a9700 1 -- 172.21.15.143:0/3086271352 shutdown_connections 2024-02-19T21:53:35.996 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:35.995+0000 7fd9a73a9700 1 -- 172.21.15.143:0/3086271352 wait complete. 2024-02-19T21:53:36.659 INFO:teuthology.orchestra.run.smithi143.stderr:Waiting for the mgr to restart... 2024-02-19T21:53:36.659 INFO:teuthology.orchestra.run.smithi143.stderr:Waiting for mgr epoch 5... 2024-02-19T21:53:36.659 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T21:53:36.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:35.993854+0000 mon.smithi143 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.143:0/3086271352' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T21:53:37.271 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.270+0000 7f4643523700 1 Processor -- start 2024-02-19T21:53:37.271 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.270+0000 7f4643523700 1 -- start start 2024-02-19T21:53:37.271 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.270+0000 7f4643523700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c0fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:37.271 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.270+0000 7f4643523700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c0fef40 con 0x7f463c070430 2024-02-19T21:53:37.271 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.270+0000 7f46412bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c0fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:37.271 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.270+0000 7f46412bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c0fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:47108/0 (socket says 172.21.15.143:47108) 2024-02-19T21:53:37.271 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.270+0000 7f46412bf700 1 -- 172.21.15.143:0/1027661967 learned_addr learned my addr 172.21.15.143:0/1027661967 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:37.271 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.270+0000 7f46412bf700 1 -- 172.21.15.143:0/1027661967 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c0ff080 con 0x7f463c070430 2024-02-19T21:53:37.271 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.271+0000 7f46412bf700 1 --2- 172.21.15.143:0/1027661967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c0fea50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f463800b5f0 tx=0x7f4638006ee0).ready entity=mon.0 client_cookie=aa24e2818e9479a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:37.271 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.271+0000 7f462bfff700 1 -- 172.21.15.143:0/1027661967 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f463800d600 con 0x7f463c070430 2024-02-19T21:53:37.272 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.271+0000 7f462bfff700 1 -- 172.21.15.143:0/1027661967 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f463800dbf0 con 0x7f463c070430 2024-02-19T21:53:37.272 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.271+0000 7f462bfff700 1 -- 172.21.15.143:0/1027661967 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4638011cb0 con 0x7f463c070430 2024-02-19T21:53:37.273 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.271+0000 7f4643523700 1 -- 172.21.15.143:0/1027661967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 msgr2=0x7f463c0fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.271+0000 7f4643523700 1 --2- 172.21.15.143:0/1027661967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c0fea50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f463800b5f0 tx=0x7f4638006ee0).stop 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.272+0000 7f4643523700 1 -- 172.21.15.143:0/1027661967 shutdown_connections 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.272+0000 7f4643523700 1 --2- 172.21.15.143:0/1027661967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c0fea50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.272+0000 7f4643523700 1 -- 172.21.15.143:0/1027661967 >> 172.21.15.143:0/1027661967 conn(0x7f463c0f6530 msgr2=0x7f463c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.272+0000 7f4643523700 1 -- 172.21.15.143:0/1027661967 shutdown_connections 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.272+0000 7f4643523700 1 -- 172.21.15.143:0/1027661967 wait complete. 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.272+0000 7f4643523700 1 Processor -- start 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.272+0000 7f4643523700 1 -- start start 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.273+0000 7f4643523700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c10ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.273+0000 7f4643523700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c10b030 con 0x7f463c070430 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.273+0000 7f46412bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c10ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.273+0000 7f46412bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c10ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:47110/0 (socket says 172.21.15.143:47110) 2024-02-19T21:53:37.274 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.273+0000 7f46412bf700 1 -- 172.21.15.143:0/2371127092 learned_addr learned my addr 172.21.15.143:0/2371127092 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:37.275 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.273+0000 7f46412bf700 1 -- 172.21.15.143:0/2371127092 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463800b200 con 0x7f463c070430 2024-02-19T21:53:37.275 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.273+0000 7f46412bf700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c10ab40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4638006800 tx=0x7f4638006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:37.275 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.273+0000 7f4629ffb700 1 -- 172.21.15.143:0/2371127092 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f463801b570 con 0x7f463c070430 2024-02-19T21:53:37.275 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.273+0000 7f4643523700 1 -- 172.21.15.143:0/2371127092 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463c10b230 con 0x7f463c070430 2024-02-19T21:53:37.275 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.273+0000 7f4629ffb700 1 -- 172.21.15.143:0/2371127092 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f463801bb60 con 0x7f463c070430 2024-02-19T21:53:37.275 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.273+0000 7f4629ffb700 1 -- 172.21.15.143:0/2371127092 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f463801a9f0 con 0x7f463c070430 2024-02-19T21:53:37.275 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.274+0000 7f4643523700 1 -- 172.21.15.143:0/2371127092 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463c10b6c0 con 0x7f463c070430 2024-02-19T21:53:37.275 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.274+0000 7f4629ffb700 1 -- 172.21.15.143:0/2371127092 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f463800f070 con 0x7f463c070430 2024-02-19T21:53:37.275 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.274+0000 7f4629ffb700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 0x7f46240322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:37.275 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.274+0000 7f4629ffb700 1 -- 172.21.15.143:0/2371127092 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4638042210 con 0x7f463c070430 2024-02-19T21:53:37.276 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.274+0000 7f4640abe700 1 -- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 msgr2=0x7f46240322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/4035514323 2024-02-19T21:53:37.276 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.274+0000 7f4640abe700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 0x7f46240322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T21:53:37.276 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.275+0000 7f4643523700 1 -- 172.21.15.143:0/2371127092 --> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f463c059100 con 0x7f462402fe10 2024-02-19T21:53:37.475 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.475+0000 7f4640abe700 1 -- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 msgr2=0x7f46240322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/4035514323 2024-02-19T21:53:37.475 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.475+0000 7f4640abe700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 0x7f46240322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T21:53:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.876+0000 7f4640abe700 1 -- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 msgr2=0x7f46240322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/4035514323 2024-02-19T21:53:37.876 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:37.876+0000 7f4640abe700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 0x7f46240322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T21:53:38.677 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:38.677+0000 7f4640abe700 1 -- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 msgr2=0x7f46240322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/4035514323 2024-02-19T21:53:38.677 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:38.677+0000 7f4640abe700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 0x7f46240322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T21:53:39.628 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:39.628+0000 7f4629ffb700 1 -- 172.21.15.143:0/2371127092 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f4638025980 con 0x7f463c070430 2024-02-19T21:53:39.628 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:39.628+0000 7f4629ffb700 1 -- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 msgr2=0x7f46240322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:53:39.628 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:39.628+0000 7f4629ffb700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 0x7f46240322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:53:39.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:39 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:39.574486+0000 mon.smithi143 (mon.0) 29 : cluster [INF] Active manager daemon smithi143.bissbc restarted 2024-02-19T21:53:39.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:39 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:39.574865+0000 mon.smithi143 (mon.0) 30 : cluster [INF] Activating manager daemon smithi143.bissbc 2024-02-19T21:53:39.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:39 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:39.579042+0000 mon.smithi143 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-19T21:53:40.638 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.637+0000 7f4629ffb700 1 -- 172.21.15.143:0/2371127092 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f4638025460 con 0x7f463c070430 2024-02-19T21:53:40.638 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.638+0000 7f4629ffb700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4624033730 0x7f4624035b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.638+0000 7f4629ffb700 1 -- 172.21.15.143:0/2371127092 --> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f463c059100 con 0x7f4624033730 2024-02-19T21:53:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.638+0000 7f4640abe700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4624033730 0x7f4624035b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.639+0000 7f4640abe700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4624033730 0x7f4624035b20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f462c0036d0 tx=0x7f462c006ca0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:40.639 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.639+0000 7f4629ffb700 1 -- 172.21.15.143:0/2371127092 <== mgr.14114 v2:172.21.15.143:6800/1142990465 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f463c059100 con 0x7f4624033730 2024-02-19T21:53:40.645 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.645+0000 7f4643523700 1 -- 172.21.15.143:0/2371127092 --> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f463c045ad0 con 0x7f4624033730 2024-02-19T21:53:40.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4629ffb700 1 -- 172.21.15.143:0/2371127092 <== mgr.14114 v2:172.21.15.143:6800/1142990465 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f463c045ad0 con 0x7f4624033730 2024-02-19T21:53:40.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: { 2024-02-19T21:53:40.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-19T21:53:40.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T21:53:40.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: } 2024-02-19T21:53:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4643523700 1 -- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4624033730 msgr2=0x7f4624035b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4643523700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4624033730 0x7f4624035b20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f462c0036d0 tx=0x7f462c006ca0).stop 2024-02-19T21:53:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4643523700 1 -- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 msgr2=0x7f463c10ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4643523700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c10ab40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4638006800 tx=0x7f4638006ee0).stop 2024-02-19T21:53:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4643523700 1 -- 172.21.15.143:0/2371127092 shutdown_connections 2024-02-19T21:53:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4643523700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4624033730 0x7f4624035b20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4643523700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f463c070430 0x7f463c10ab40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4643523700 1 --2- 172.21.15.143:0/2371127092 >> [v2:172.21.15.143:6800/4035514323,v1:172.21.15.143:6801/4035514323] conn(0x7f462402fe10 0x7f46240322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:53:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4643523700 1 -- 172.21.15.143:0/2371127092 >> 172.21.15.143:0/2371127092 conn(0x7f463c0f6530 msgr2=0x7f463c0f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4643523700 1 -- 172.21.15.143:0/2371127092 shutdown_connections 2024-02-19T21:53:40.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:40.646+0000 7f4643523700 1 -- 172.21.15.143:0/2371127092 wait complete. 2024-02-19T21:53:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:39.630441+0000 mon.smithi143 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi143.bissbc(active, starting, since 0.0556566s) 2024-02-19T21:53:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.632967+0000 mon.smithi143 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.633606+0000 mon.smithi143 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mgr metadata", "who": "smithi143.bissbc", "id": "smithi143.bissbc"}]: dispatch 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.634284+0000 mon.smithi143 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.634478+0000 mon.smithi143 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.634660+0000 mon.smithi143 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:39.648144+0000 mon.smithi143 (mon.0) 38 : cluster [INF] Manager daemon smithi143.bissbc is now available 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.661019+0000 mon.smithi143 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.663811+0000 mon.smithi143 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.666321+0000 mon.smithi143 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.754913+0000 mon.smithi143 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.755549+0000 mon.smithi143 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.757118+0000 mon.smithi143 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.759272+0000 mon.smithi143 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:53:40.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.768484+0000 mon.smithi143 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:53:40.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:39.769615+0000 mon.smithi143 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T21:53:41.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:41 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:40.639980+0000 mgr.smithi143.bissbc (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T21:53:41.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:41 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:40.640562+0000 mon.smithi143 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi143.bissbc(active, since 1.06577s) 2024-02-19T21:53:41.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:41 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:40.646755+0000 mgr.smithi143.bissbc (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T21:53:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:53:41.642804+0000 mon.smithi143 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi143.bissbc(active, since 2s) 2024-02-19T21:53:46.085 INFO:teuthology.orchestra.run.smithi143.stderr:mgr epoch 5 is available 2024-02-19T21:53:46.086 INFO:teuthology.orchestra.run.smithi143.stderr:Setting orchestrator backend to cephadm... 2024-02-19T21:53:46.086 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-19T21:53:47.885 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.881+0000 7f2c9d7de700 1 Processor -- start 2024-02-19T21:53:47.885 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.881+0000 7f2c9d7de700 1 -- start start 2024-02-19T21:53:47.885 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.881+0000 7f2c9d7de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.881+0000 7f2c9d7de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c980fef80 con 0x7f2c980703f0 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.882+0000 7f2c96ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.882+0000 7f2c96ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35918/0 (socket says 172.21.15.143:35918) 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.882+0000 7f2c96ffd700 1 -- 172.21.15.143:0/1469676745 learned_addr learned my addr 172.21.15.143:0/1469676745 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.882+0000 7f2c96ffd700 1 -- 172.21.15.143:0/1469676745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c980ff0c0 con 0x7f2c980703f0 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.882+0000 7f2c96ffd700 1 --2- 172.21.15.143:0/1469676745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fea90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2c8c00b5f0 tx=0x7f2c8c006ee0).ready entity=mon.0 client_cookie=30ec92e8b3902630 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.882+0000 7f2c95ffb700 1 -- 172.21.15.143:0/1469676745 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c8c00d600 con 0x7f2c980703f0 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.882+0000 7f2c95ffb700 1 -- 172.21.15.143:0/1469676745 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2c8c00dbf0 con 0x7f2c980703f0 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.882+0000 7f2c95ffb700 1 -- 172.21.15.143:0/1469676745 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c8c011ce0 con 0x7f2c980703f0 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.883+0000 7f2c9d7de700 1 -- 172.21.15.143:0/1469676745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 msgr2=0x7f2c980fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.883+0000 7f2c9d7de700 1 --2- 172.21.15.143:0/1469676745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fea90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2c8c00b5f0 tx=0x7f2c8c006ee0).stop 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.883+0000 7f2c9d7de700 1 -- 172.21.15.143:0/1469676745 shutdown_connections 2024-02-19T21:53:47.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.883+0000 7f2c9d7de700 1 --2- 172.21.15.143:0/1469676745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fea90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:47.887 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.883+0000 7f2c9d7de700 1 -- 172.21.15.143:0/1469676745 >> 172.21.15.143:0/1469676745 conn(0x7f2c980f6530 msgr2=0x7f2c980f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:47.887 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.883+0000 7f2c9d7de700 1 -- 172.21.15.143:0/1469676745 shutdown_connections 2024-02-19T21:53:47.887 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.883+0000 7f2c9d7de700 1 -- 172.21.15.143:0/1469676745 wait complete. 2024-02-19T21:53:47.887 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.884+0000 7f2c9d7de700 1 Processor -- start 2024-02-19T21:53:47.887 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.884+0000 7f2c9d7de700 1 -- start start 2024-02-19T21:53:47.887 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.884+0000 7f2c9d7de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:47.887 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.884+0000 7f2c9d7de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c980fa990 con 0x7f2c980703f0 2024-02-19T21:53:47.887 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.884+0000 7f2c96ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:47.887 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.884+0000 7f2c96ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35920/0 (socket says 172.21.15.143:35920) 2024-02-19T21:53:47.887 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.884+0000 7f2c96ffd700 1 -- 172.21.15.143:0/4068710050 learned_addr learned my addr 172.21.15.143:0/4068710050 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:47.888 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.884+0000 7f2c96ffd700 1 -- 172.21.15.143:0/4068710050 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c00b200 con 0x7f2c980703f0 2024-02-19T21:53:47.888 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.884+0000 7f2c96ffd700 1 --2- 172.21.15.143:0/4068710050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fc510 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f2c8c011ed0 tx=0x7f2c8c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:47.888 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.884+0000 7f2c7f7fe700 1 -- 172.21.15.143:0/4068710050 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c8c01b570 con 0x7f2c980703f0 2024-02-19T21:53:47.888 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.884+0000 7f2c9d7de700 1 -- 172.21.15.143:0/4068710050 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c980fab30 con 0x7f2c980703f0 2024-02-19T21:53:47.888 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.885+0000 7f2c7f7fe700 1 -- 172.21.15.143:0/4068710050 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2c8c01bb60 con 0x7f2c980703f0 2024-02-19T21:53:47.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.885+0000 7f2c7f7fe700 1 -- 172.21.15.143:0/4068710050 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c8c01aad0 con 0x7f2c980703f0 2024-02-19T21:53:47.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.885+0000 7f2c9d7de700 1 -- 172.21.15.143:0/4068710050 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c980fafc0 con 0x7f2c980703f0 2024-02-19T21:53:47.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.885+0000 7f2c7f7fe700 1 -- 172.21.15.143:0/4068710050 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f2c8c00f070 con 0x7f2c980703f0 2024-02-19T21:53:47.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.885+0000 7f2c9d7de700 1 -- 172.21.15.143:0/4068710050 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c98045ad0 con 0x7f2c980703f0 2024-02-19T21:53:47.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.886+0000 7f2c7f7fe700 1 --2- 172.21.15.143:0/4068710050 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f2c8402fe10 0x7f2c840322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:47.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.886+0000 7f2c7f7fe700 1 -- 172.21.15.143:0/4068710050 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2c8c0413b0 con 0x7f2c980703f0 2024-02-19T21:53:47.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.886+0000 7f2c967fc700 1 --2- 172.21.15.143:0/4068710050 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f2c8402fe10 0x7f2c840322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:47.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.886+0000 7f2c967fc700 1 --2- 172.21.15.143:0/4068710050 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f2c8402fe10 0x7f2c840322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2c880096a0 tx=0x7f2c88006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:47.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:47.890+0000 7f2c7f7fe700 1 -- 172.21.15.143:0/4068710050 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c8c010360 con 0x7f2c980703f0 2024-02-19T21:53:48.069 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.068+0000 7f2c9d7de700 1 -- 172.21.15.143:0/4068710050 --> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c980fff80 con 0x7f2c8402fe10 2024-02-19T21:53:48.074 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.074+0000 7f2c7f7fe700 1 -- 172.21.15.143:0/4068710050 <== mgr.14114 v2:172.21.15.143:6800/1142990465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2c980fff80 con 0x7f2c8402fe10 2024-02-19T21:53:48.077 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.076+0000 7f2c9d7de700 1 -- 172.21.15.143:0/4068710050 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f2c8402fe10 msgr2=0x7f2c840322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:48.077 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.076+0000 7f2c9d7de700 1 --2- 172.21.15.143:0/4068710050 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f2c8402fe10 0x7f2c840322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2c880096a0 tx=0x7f2c88006b40).stop 2024-02-19T21:53:48.077 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.076+0000 7f2c9d7de700 1 -- 172.21.15.143:0/4068710050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 msgr2=0x7f2c980fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:48.077 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.076+0000 7f2c9d7de700 1 --2- 172.21.15.143:0/4068710050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fc510 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f2c8c011ed0 tx=0x7f2c8c006ee0).stop 2024-02-19T21:53:48.078 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.076+0000 7f2c9d7de700 1 -- 172.21.15.143:0/4068710050 shutdown_connections 2024-02-19T21:53:48.078 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.076+0000 7f2c9d7de700 1 --2- 172.21.15.143:0/4068710050 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f2c8402fe10 0x7f2c840322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:48.078 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.076+0000 7f2c9d7de700 1 --2- 172.21.15.143:0/4068710050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c980703f0 0x7f2c980fc510 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:48.078 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.076+0000 7f2c9d7de700 1 -- 172.21.15.143:0/4068710050 >> 172.21.15.143:0/4068710050 conn(0x7f2c980f6530 msgr2=0x7f2c980f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:48.078 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.076+0000 7f2c9d7de700 1 -- 172.21.15.143:0/4068710050 shutdown_connections 2024-02-19T21:53:48.078 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:48.076+0000 7f2c9d7de700 1 -- 172.21.15.143:0/4068710050 wait complete. 2024-02-19T21:53:48.698 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-19T21:53:49.211 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.208+0000 7f3f1ff8e700 1 Processor -- start 2024-02-19T21:53:49.211 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.209+0000 7f3f1ff8e700 1 -- start start 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.209+0000 7f3f1ff8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.209+0000 7f3f1ff8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180fef40 con 0x7f3f18070430 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.209+0000 7f3f1dd2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.209+0000 7f3f1dd2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35936/0 (socket says 172.21.15.143:35936) 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.209+0000 7f3f1dd2a700 1 -- 172.21.15.143:0/2432086455 learned_addr learned my addr 172.21.15.143:0/2432086455 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.209+0000 7f3f1dd2a700 1 -- 172.21.15.143:0/2432086455 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180ff080 con 0x7f3f18070430 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.209+0000 7f3f1dd2a700 1 --2- 172.21.15.143:0/2432086455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fea50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3f0800b5f0 tx=0x7f3f08006ee0).ready entity=mon.0 client_cookie=effe66618df144fe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.209+0000 7f3f1cd28700 1 -- 172.21.15.143:0/2432086455 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0800d600 con 0x7f3f18070430 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.210+0000 7f3f1cd28700 1 -- 172.21.15.143:0/2432086455 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3f0800dbf0 con 0x7f3f18070430 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.210+0000 7f3f1cd28700 1 -- 172.21.15.143:0/2432086455 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f08011cb0 con 0x7f3f18070430 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.210+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/2432086455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 msgr2=0x7f3f180fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.210+0000 7f3f1ff8e700 1 --2- 172.21.15.143:0/2432086455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fea50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3f0800b5f0 tx=0x7f3f08006ee0).stop 2024-02-19T21:53:49.212 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.210+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/2432086455 shutdown_connections 2024-02-19T21:53:49.213 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.210+0000 7f3f1ff8e700 1 --2- 172.21.15.143:0/2432086455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fea50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:49.213 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.210+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/2432086455 >> 172.21.15.143:0/2432086455 conn(0x7f3f180f6530 msgr2=0x7f3f180f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:49.213 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.210+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/2432086455 shutdown_connections 2024-02-19T21:53:49.213 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.210+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/2432086455 wait complete. 2024-02-19T21:53:49.215 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.211+0000 7f3f1ff8e700 1 Processor -- start 2024-02-19T21:53:49.215 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.211+0000 7f3f1ff8e700 1 -- start start 2024-02-19T21:53:49.215 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.211+0000 7f3f1ff8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.211+0000 7f3f1ff8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180fa950 con 0x7f3f18070430 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.211+0000 7f3f1dd2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.211+0000 7f3f1dd2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35950/0 (socket says 172.21.15.143:35950) 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.211+0000 7f3f1dd2a700 1 -- 172.21.15.143:0/1577952172 learned_addr learned my addr 172.21.15.143:0/1577952172 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.212+0000 7f3f1dd2a700 1 -- 172.21.15.143:0/1577952172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0800b200 con 0x7f3f18070430 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.212+0000 7f3f1dd2a700 1 --2- 172.21.15.143:0/1577952172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fc4d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3f08006800 tx=0x7f3f08006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.212+0000 7f3f067fc700 1 -- 172.21.15.143:0/1577952172 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0801b570 con 0x7f3f18070430 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.212+0000 7f3f067fc700 1 -- 172.21.15.143:0/1577952172 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3f0801bb60 con 0x7f3f18070430 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.212+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/1577952172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f180faaf0 con 0x7f3f18070430 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.212+0000 7f3f067fc700 1 -- 172.21.15.143:0/1577952172 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0801a9f0 con 0x7f3f18070430 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.212+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/1577952172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f180faf80 con 0x7f3f18070430 2024-02-19T21:53:49.216 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.212+0000 7f3f067fc700 1 -- 172.21.15.143:0/1577952172 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f3f0800f070 con 0x7f3f18070430 2024-02-19T21:53:49.217 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.213+0000 7f3f067fc700 1 --2- 172.21.15.143:0/1577952172 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f3f0002fe10 0x7f3f000322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:49.217 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.213+0000 7f3f067fc700 1 -- 172.21.15.143:0/1577952172 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3f08041590 con 0x7f3f18070430 2024-02-19T21:53:49.217 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.213+0000 7f3f1d529700 1 --2- 172.21.15.143:0/1577952172 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f3f0002fe10 0x7f3f000322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:49.217 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.213+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/1577952172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f18059100 con 0x7f3f18070430 2024-02-19T21:53:49.217 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.213+0000 7f3f1d529700 1 --2- 172.21.15.143:0/1577952172 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f3f0002fe10 0x7f3f000322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3f140096a0 tx=0x7f3f14006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:49.218 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.217+0000 7f3f067fc700 1 -- 172.21.15.143:0/1577952172 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f08016070 con 0x7f3f18070430 2024-02-19T21:53:49.367 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.366+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/1577952172 --> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f18104af0 con 0x7f3f0002fe10 2024-02-19T21:53:49.368 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.367+0000 7f3f067fc700 1 -- 172.21.15.143:0/1577952172 <== mgr.14114 v2:172.21.15.143:6800/1142990465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f3f18104af0 con 0x7f3f0002fe10 2024-02-19T21:53:49.368 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: value unchanged 2024-02-19T21:53:49.370 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.369+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/1577952172 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f3f0002fe10 msgr2=0x7f3f000322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:49.370 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.369+0000 7f3f1ff8e700 1 --2- 172.21.15.143:0/1577952172 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f3f0002fe10 0x7f3f000322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3f140096a0 tx=0x7f3f14006b40).stop 2024-02-19T21:53:49.370 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.369+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/1577952172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 msgr2=0x7f3f180fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:49.370 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.369+0000 7f3f1ff8e700 1 --2- 172.21.15.143:0/1577952172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fc4d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3f08006800 tx=0x7f3f08006ee0).stop 2024-02-19T21:53:49.370 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.369+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/1577952172 shutdown_connections 2024-02-19T21:53:49.370 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.369+0000 7f3f1ff8e700 1 --2- 172.21.15.143:0/1577952172 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f3f0002fe10 0x7f3f000322d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.369+0000 7f3f1ff8e700 1 --2- 172.21.15.143:0/1577952172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18070430 0x7f3f180fc4d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.369+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/1577952172 >> 172.21.15.143:0/1577952172 conn(0x7f3f180f6530 msgr2=0x7f3f180f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.369+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/1577952172 shutdown_connections 2024-02-19T21:53:49.371 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:49.369+0000 7f3f1ff8e700 1 -- 172.21.15.143:0/1577952172 wait complete. 2024-02-19T21:53:49.395 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:49 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:48.071479+0000 mgr.smithi143.bissbc (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:53:49.395 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:49 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:48.075470+0000 mon.smithi143 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:49.396 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:49 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:48.079126+0000 mon.smithi143 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:53:50.004 INFO:teuthology.orchestra.run.smithi143.stderr:Generating ssh key... 2024-02-19T21:53:50.004 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-19T21:53:50.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:49.369022+0000 mgr.smithi143.bissbc (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:53:50.533 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.531+0000 7ff965770700 1 Processor -- start 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.531+0000 7ff965770700 1 -- start start 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.531+0000 7ff965770700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff9600fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.531+0000 7ff965770700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600fc7b0 con 0x7ff9600fbea0 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.531+0000 7ff95effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff9600fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.531+0000 7ff95effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff9600fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40716/0 (socket says 172.21.15.143:40716) 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.531+0000 7ff95effd700 1 -- 172.21.15.143:0/3676049469 learned_addr learned my addr 172.21.15.143:0/3676049469 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.531+0000 7ff95effd700 1 -- 172.21.15.143:0/3676049469 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9600fc8f0 con 0x7ff9600fbea0 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.531+0000 7ff95effd700 1 --2- 172.21.15.143:0/3676049469 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff9600fc2c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff95000b5f0 tx=0x7ff950006ee0).ready entity=mon.0 client_cookie=8527982f40545eb6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.532+0000 7ff95dffb700 1 -- 172.21.15.143:0/3676049469 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff95000d600 con 0x7ff9600fbea0 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.532+0000 7ff95dffb700 1 -- 172.21.15.143:0/3676049469 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff95000dbf0 con 0x7ff9600fbea0 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.532+0000 7ff95dffb700 1 -- 172.21.15.143:0/3676049469 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff950011cb0 con 0x7ff9600fbea0 2024-02-19T21:53:50.534 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.532+0000 7ff965770700 1 -- 172.21.15.143:0/3676049469 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 msgr2=0x7ff9600fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:50.535 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.532+0000 7ff965770700 1 --2- 172.21.15.143:0/3676049469 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff9600fc2c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff95000b5f0 tx=0x7ff950006ee0).stop 2024-02-19T21:53:50.535 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.532+0000 7ff965770700 1 -- 172.21.15.143:0/3676049469 shutdown_connections 2024-02-19T21:53:50.535 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.532+0000 7ff965770700 1 --2- 172.21.15.143:0/3676049469 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff9600fc2c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:50.535 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.532+0000 7ff965770700 1 -- 172.21.15.143:0/3676049469 >> 172.21.15.143:0/3676049469 conn(0x7ff9600f76a0 msgr2=0x7ff9600f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:50.535 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.532+0000 7ff965770700 1 -- 172.21.15.143:0/3676049469 shutdown_connections 2024-02-19T21:53:50.535 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.532+0000 7ff965770700 1 -- 172.21.15.143:0/3676049469 wait complete. 2024-02-19T21:53:50.535 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.533+0000 7ff965770700 1 Processor -- start 2024-02-19T21:53:50.535 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.533+0000 7ff965770700 1 -- start start 2024-02-19T21:53:50.536 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.533+0000 7ff965770700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff96010ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.533+0000 7ff965770700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96010b080 con 0x7ff9600fbea0 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.533+0000 7ff95effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff96010ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.533+0000 7ff95effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff96010ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40726/0 (socket says 172.21.15.143:40726) 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.534+0000 7ff95effd700 1 -- 172.21.15.143:0/2875353169 learned_addr learned my addr 172.21.15.143:0/2875353169 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.534+0000 7ff95effd700 1 -- 172.21.15.143:0/2875353169 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95000b200 con 0x7ff9600fbea0 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.534+0000 7ff95effd700 1 --2- 172.21.15.143:0/2875353169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff96010ab90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff95000bf40 tx=0x7ff950006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.534+0000 7ff9477fe700 1 -- 172.21.15.143:0/2875353169 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff95001b570 con 0x7ff9600fbea0 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.534+0000 7ff9477fe700 1 -- 172.21.15.143:0/2875353169 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff95001bb60 con 0x7ff9600fbea0 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.534+0000 7ff965770700 1 -- 172.21.15.143:0/2875353169 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96010b280 con 0x7ff9600fbea0 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.534+0000 7ff9477fe700 1 -- 172.21.15.143:0/2875353169 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff95001aa20 con 0x7ff9600fbea0 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.534+0000 7ff965770700 1 -- 172.21.15.143:0/2875353169 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96010b710 con 0x7ff9600fbea0 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.535+0000 7ff9477fe700 1 -- 172.21.15.143:0/2875353169 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff95000f070 con 0x7ff9600fbea0 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.535+0000 7ff9477fe700 1 --2- 172.21.15.143:0/2875353169 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7ff94c02fe70 0x7ff94c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.535+0000 7ff9477fe700 1 -- 172.21.15.143:0/2875353169 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff950041400 con 0x7ff9600fbea0 2024-02-19T21:53:50.537 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.535+0000 7ff95e7fc700 1 --2- 172.21.15.143:0/2875353169 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7ff94c02fe70 0x7ff94c032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:50.538 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.535+0000 7ff95e7fc700 1 --2- 172.21.15.143:0/2875353169 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7ff94c02fe70 0x7ff94c032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff9540096a0 tx=0x7ff954006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:50.538 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.535+0000 7ff965770700 1 -- 172.21.15.143:0/2875353169 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff960045ad0 con 0x7ff9600fbea0 2024-02-19T21:53:50.540 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.540+0000 7ff9477fe700 1 -- 172.21.15.143:0/2875353169 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff950016070 con 0x7ff9600fbea0 2024-02-19T21:53:50.691 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.690+0000 7ff965770700 1 -- 172.21.15.143:0/2875353169 --> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff960104c80 con 0x7ff94c02fe70 2024-02-19T21:53:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.873+0000 7ff9477fe700 1 -- 172.21.15.143:0/2875353169 <== mgr.14114 v2:172.21.15.143:6800/1142990465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff960104c80 con 0x7ff94c02fe70 2024-02-19T21:53:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.874+0000 7ff965770700 1 -- 172.21.15.143:0/2875353169 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7ff94c02fe70 msgr2=0x7ff94c032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.874+0000 7ff965770700 1 --2- 172.21.15.143:0/2875353169 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7ff94c02fe70 0x7ff94c032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff9540096a0 tx=0x7ff954006b40).stop 2024-02-19T21:53:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.874+0000 7ff965770700 1 -- 172.21.15.143:0/2875353169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 msgr2=0x7ff96010ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.874+0000 7ff965770700 1 --2- 172.21.15.143:0/2875353169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff96010ab90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff95000bf40 tx=0x7ff950006ee0).stop 2024-02-19T21:53:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.874+0000 7ff965770700 1 -- 172.21.15.143:0/2875353169 shutdown_connections 2024-02-19T21:53:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.874+0000 7ff965770700 1 --2- 172.21.15.143:0/2875353169 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7ff94c02fe70 0x7ff94c032330 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.874+0000 7ff965770700 1 --2- 172.21.15.143:0/2875353169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9600fbea0 0x7ff96010ab90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.874+0000 7ff965770700 1 -- 172.21.15.143:0/2875353169 >> 172.21.15.143:0/2875353169 conn(0x7ff9600f76a0 msgr2=0x7ff9600f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.875+0000 7ff965770700 1 -- 172.21.15.143:0/2875353169 shutdown_connections 2024-02-19T21:53:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:50.875+0000 7ff965770700 1 -- 172.21.15.143:0/2875353169 wait complete. 2024-02-19T21:53:51.467 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-19T21:53:52.032 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.031+0000 7f4eb8305700 1 Processor -- start 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.031+0000 7f4eb8305700 1 -- start start 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.031+0000 7f4eb8305700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb00fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.031+0000 7f4eb8305700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb00fb6a0 con 0x7f4eb00fad90 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.032+0000 7f4eb60a1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb00fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.032+0000 7f4eb60a1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb00fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40730/0 (socket says 172.21.15.143:40730) 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.032+0000 7f4eb60a1700 1 -- 172.21.15.143:0/3066671130 learned_addr learned my addr 172.21.15.143:0/3066671130 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.032+0000 7f4eb60a1700 1 -- 172.21.15.143:0/3066671130 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb00fb7e0 con 0x7f4eb00fad90 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.032+0000 7f4eb60a1700 1 --2- 172.21.15.143:0/3066671130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb00fb1b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4ea000b5f0 tx=0x7f4ea0006ee0).ready entity=mon.0 client_cookie=3a163cc2ab861840 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.032+0000 7f4eb509f700 1 -- 172.21.15.143:0/3066671130 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ea000d600 con 0x7f4eb00fad90 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.032+0000 7f4eb509f700 1 -- 172.21.15.143:0/3066671130 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4ea000dbf0 con 0x7f4eb00fad90 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.032+0000 7f4eb509f700 1 -- 172.21.15.143:0/3066671130 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ea0011cb0 con 0x7f4eb00fad90 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.033+0000 7f4eb8305700 1 -- 172.21.15.143:0/3066671130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 msgr2=0x7f4eb00fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:52.034 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.033+0000 7f4eb8305700 1 --2- 172.21.15.143:0/3066671130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb00fb1b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4ea000b5f0 tx=0x7f4ea0006ee0).stop 2024-02-19T21:53:52.037 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.033+0000 7f4eb8305700 1 -- 172.21.15.143:0/3066671130 shutdown_connections 2024-02-19T21:53:52.037 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.033+0000 7f4eb8305700 1 --2- 172.21.15.143:0/3066671130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb00fb1b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:52.037 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.033+0000 7f4eb8305700 1 -- 172.21.15.143:0/3066671130 >> 172.21.15.143:0/3066671130 conn(0x7f4eb00f6530 msgr2=0x7f4eb00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.033+0000 7f4eb8305700 1 -- 172.21.15.143:0/3066671130 shutdown_connections 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.033+0000 7f4eb8305700 1 -- 172.21.15.143:0/3066671130 wait complete. 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.033+0000 7f4eb8305700 1 Processor -- start 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4eb8305700 1 -- start start 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4eb8305700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb0109a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4eb8305700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb0109f30 con 0x7f4eb00fad90 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4eb60a1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb0109a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4eb60a1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb0109a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40736/0 (socket says 172.21.15.143:40736) 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4eb60a1700 1 -- 172.21.15.143:0/3270647523 learned_addr learned my addr 172.21.15.143:0/3270647523 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4eb60a1700 1 -- 172.21.15.143:0/3270647523 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea000b200 con 0x7f4eb00fad90 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4eb60a1700 1 --2- 172.21.15.143:0/3270647523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb0109a40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4ea0011ea0 tx=0x7f4ea0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4e9effd700 1 -- 172.21.15.143:0/3270647523 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ea001b570 con 0x7f4eb00fad90 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4e9effd700 1 -- 172.21.15.143:0/3270647523 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4ea001bb60 con 0x7f4eb00fad90 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4eb8305700 1 -- 172.21.15.143:0/3270647523 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb010a130 con 0x7f4eb00fad90 2024-02-19T21:53:52.038 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.034+0000 7f4e9effd700 1 -- 172.21.15.143:0/3270647523 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ea001a8d0 con 0x7f4eb00fad90 2024-02-19T21:53:52.039 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.035+0000 7f4eb8305700 1 -- 172.21.15.143:0/3270647523 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb010a5c0 con 0x7f4eb00fad90 2024-02-19T21:53:52.039 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.035+0000 7f4e9effd700 1 -- 172.21.15.143:0/3270647523 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4ea000f070 con 0x7f4eb00fad90 2024-02-19T21:53:52.039 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.035+0000 7f4e9effd700 1 --2- 172.21.15.143:0/3270647523 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4e9802fe70 0x7f4e98032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:52.039 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.035+0000 7f4e9effd700 1 -- 172.21.15.143:0/3270647523 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4ea00413e0 con 0x7f4eb00fad90 2024-02-19T21:53:52.039 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.035+0000 7f4eb58a0700 1 --2- 172.21.15.143:0/3270647523 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4e9802fe70 0x7f4e98032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:52.039 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.036+0000 7f4eb58a0700 1 --2- 172.21.15.143:0/3270647523 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4e9802fe70 0x7f4e98032330 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4eac0096a0 tx=0x7f4eac006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:52.039 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.036+0000 7f4eb8305700 1 -- 172.21.15.143:0/3270647523 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb0045ad0 con 0x7f4eb00fad90 2024-02-19T21:53:52.043 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.041+0000 7f4e9effd700 1 -- 172.21.15.143:0/3270647523 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ea0016070 con 0x7f4eb00fad90 2024-02-19T21:53:52.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:51 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:53:50.692947+0000 mgr.smithi143.bissbc (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-19T21:53:52.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:51 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:50.872232+0000 mon.smithi143 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:52.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:51 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:50.873903+0000 mon.smithi143 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:52.190 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.190+0000 7f4eb8305700 1 -- 172.21.15.143:0/3270647523 --> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4eb00008d0 con 0x7f4e9802fe70 2024-02-19T21:53:52.191 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.190+0000 7f4e9effd700 1 -- 172.21.15.143:0/3270647523 <== mgr.14114 v2:172.21.15.143:6800/1142990465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f4eb00008d0 con 0x7f4e9802fe70 2024-02-19T21:53:52.191 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvuJIirlmehmbHme0GOsCEKht/SPbbEWBAQAooTY3BYoiMyYX3+8HvAiDLl0PNh17Hd6C9TaiNJZs/XiqM+uDdHECEln6feGQ3hfNE9RUGaxSciJVLIrwHEI+RjD+u/Le7qcqFjseH6Zv1jJLJN1CIe86hDYKbvmKYZOiEHI0UZGSeHHNPbnTDUkO7TfefMc2DiaW0qh7RYDUzbvqE9AUqX2f+Ln1vrReh5bAmNmeigfsiTtqSjcC3RGV3BD9QECEnOwLNmVZsQn8B0oqb9sHy1IgVRAspa2P+YoX96ecFhTUE25BhuPPPl+J5FZv/rVAAQmbw30ssXAcKwZKP6jvKVj6leGA+ljzkx/pfKz+95dQgUta/CF5oKbDvGhEYFcNvN1z+9bweBgJP+BMHk3eFvALliud2hFigpuv8hovA8LmeAx/85GZ7YZOsiNEhvcgRDQzxgobZCg+PGgrAam1VGxSVDr+9TbrRb2aKeeRUO5wawg3lrQ9RdQQop262/Nc= ceph-38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:53:52.193 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.192+0000 7f4eb8305700 1 -- 172.21.15.143:0/3270647523 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4e9802fe70 msgr2=0x7f4e98032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:52.193 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.192+0000 7f4eb8305700 1 --2- 172.21.15.143:0/3270647523 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4e9802fe70 0x7f4e98032330 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4eac0096a0 tx=0x7f4eac006b40).stop 2024-02-19T21:53:52.193 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.192+0000 7f4eb8305700 1 -- 172.21.15.143:0/3270647523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 msgr2=0x7f4eb0109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:52.194 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.192+0000 7f4eb8305700 1 --2- 172.21.15.143:0/3270647523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb0109a40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4ea0011ea0 tx=0x7f4ea0006ee0).stop 2024-02-19T21:53:52.194 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.192+0000 7f4eb8305700 1 -- 172.21.15.143:0/3270647523 shutdown_connections 2024-02-19T21:53:52.194 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.192+0000 7f4eb8305700 1 --2- 172.21.15.143:0/3270647523 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f4e9802fe70 0x7f4e98032330 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:52.194 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.192+0000 7f4eb8305700 1 --2- 172.21.15.143:0/3270647523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb00fad90 0x7f4eb0109a40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:52.194 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.192+0000 7f4eb8305700 1 -- 172.21.15.143:0/3270647523 >> 172.21.15.143:0/3270647523 conn(0x7f4eb00f6530 msgr2=0x7f4eb00f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:52.194 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.192+0000 7f4eb8305700 1 -- 172.21.15.143:0/3270647523 shutdown_connections 2024-02-19T21:53:52.194 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:52.192+0000 7f4eb8305700 1 -- 172.21.15.143:0/3270647523 wait complete. 2024-02-19T21:53:52.799 INFO:teuthology.orchestra.run.smithi143.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-19T21:53:52.799 INFO:teuthology.orchestra.run.smithi143.stderr:Adding key to root@localhost authorized_keys... 2024-02-19T21:53:52.800 INFO:teuthology.orchestra.run.smithi143.stderr:Adding host smithi143... 2024-02-19T21:53:52.800 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi143 172.21.15.143 2024-02-19T21:53:53.444 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.440+0000 7fb207448700 1 Processor -- start 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.440+0000 7fb207448700 1 -- start start 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.440+0000 7fb207448700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.440+0000 7fb207448700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2000fc7b0 con 0x7fb2000fbea0 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.441+0000 7fb2051e4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.441+0000 7fb2051e4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40748/0 (socket says 172.21.15.143:40748) 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.441+0000 7fb2051e4700 1 -- 172.21.15.143:0/1034262389 learned_addr learned my addr 172.21.15.143:0/1034262389 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.441+0000 7fb2051e4700 1 -- 172.21.15.143:0/1034262389 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2000fc8f0 con 0x7fb2000fbea0 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.441+0000 7fb2051e4700 1 --2- 172.21.15.143:0/1034262389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000fc2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb1f000b5f0 tx=0x7fb1f0006ee0).ready entity=mon.0 client_cookie=b3bc465c4c7071cb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.441+0000 7fb1effff700 1 -- 172.21.15.143:0/1034262389 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f000d600 con 0x7fb2000fbea0 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.441+0000 7fb1effff700 1 -- 172.21.15.143:0/1034262389 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb1f000dbf0 con 0x7fb2000fbea0 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.441+0000 7fb1effff700 1 -- 172.21.15.143:0/1034262389 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f0011cb0 con 0x7fb2000fbea0 2024-02-19T21:53:53.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.442+0000 7fb207448700 1 -- 172.21.15.143:0/1034262389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 msgr2=0x7fb2000fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.442+0000 7fb207448700 1 --2- 172.21.15.143:0/1034262389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000fc2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb1f000b5f0 tx=0x7fb1f0006ee0).stop 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.442+0000 7fb207448700 1 -- 172.21.15.143:0/1034262389 shutdown_connections 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.442+0000 7fb207448700 1 --2- 172.21.15.143:0/1034262389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000fc2c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.442+0000 7fb207448700 1 -- 172.21.15.143:0/1034262389 >> 172.21.15.143:0/1034262389 conn(0x7fb2000f76a0 msgr2=0x7fb2000f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.442+0000 7fb207448700 1 -- 172.21.15.143:0/1034262389 shutdown_connections 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.442+0000 7fb207448700 1 -- 172.21.15.143:0/1034262389 wait complete. 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.443+0000 7fb207448700 1 Processor -- start 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.443+0000 7fb207448700 1 -- start start 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.443+0000 7fb207448700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.443+0000 7fb207448700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200070a60 con 0x7fb2000fbea0 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.443+0000 7fb2051e4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.443+0000 7fb2051e4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40760/0 (socket says 172.21.15.143:40760) 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.443+0000 7fb2051e4700 1 -- 172.21.15.143:0/318984232 learned_addr learned my addr 172.21.15.143:0/318984232 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.443+0000 7fb2051e4700 1 -- 172.21.15.143:0/318984232 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f000b200 con 0x7fb2000fbea0 2024-02-19T21:53:53.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.443+0000 7fb2051e4700 1 --2- 172.21.15.143:0/318984232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000725e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb1f0002f40 tx=0x7fb1f0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:53.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.443+0000 7fb1edffb700 1 -- 172.21.15.143:0/318984232 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f001b410 con 0x7fb2000fbea0 2024-02-19T21:53:53.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.444+0000 7fb1edffb700 1 -- 172.21.15.143:0/318984232 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb1f001ba00 con 0x7fb2000fbea0 2024-02-19T21:53:53.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.444+0000 7fb1edffb700 1 -- 172.21.15.143:0/318984232 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f001a980 con 0x7fb2000fbea0 2024-02-19T21:53:53.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.444+0000 7fb207448700 1 -- 172.21.15.143:0/318984232 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb200070c00 con 0x7fb2000fbea0 2024-02-19T21:53:53.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.444+0000 7fb207448700 1 -- 172.21.15.143:0/318984232 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb200071090 con 0x7fb2000fbea0 2024-02-19T21:53:53.448 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.444+0000 7fb1edffb700 1 -- 172.21.15.143:0/318984232 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb1f000f070 con 0x7fb2000fbea0 2024-02-19T21:53:53.448 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.445+0000 7fb1edffb700 1 --2- 172.21.15.143:0/318984232 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb1e802fe70 0x7fb1e8032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:53.448 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.445+0000 7fb1edffb700 1 -- 172.21.15.143:0/318984232 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb1f0041390 con 0x7fb2000fbea0 2024-02-19T21:53:53.448 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.445+0000 7fb207448700 1 -- 172.21.15.143:0/318984232 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb200045ad0 con 0x7fb2000fbea0 2024-02-19T21:53:53.448 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.445+0000 7fb2049e3700 1 --2- 172.21.15.143:0/318984232 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb1e802fe70 0x7fb1e8032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:53.448 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.445+0000 7fb2049e3700 1 --2- 172.21.15.143:0/318984232 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb1e802fe70 0x7fb1e8032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb1fc0096a0 tx=0x7fb1fc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:53.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.450+0000 7fb1edffb700 1 -- 172.21.15.143:0/318984232 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1f0016070 con 0x7fb2000fbea0 2024-02-19T21:53:53.620 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:53.619+0000 7fb207448700 1 -- 172.21.15.143:0/318984232 --> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi143", "addr": "172.21.15.143", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2000fdce0 con 0x7fb1e802fe70 2024-02-19T21:53:53.720 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:53 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:52.192451+0000 mgr.smithi143.bissbc (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:53:54.223 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: Added host 'smithi143' 2024-02-19T21:53:54.223 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:54.221+0000 7fb1edffb700 1 -- 172.21.15.143:0/318984232 <== mgr.14114 v2:172.21.15.143:6800/1142990465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fb2000fdce0 con 0x7fb1e802fe70 2024-02-19T21:53:54.225 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:54.224+0000 7fb207448700 1 -- 172.21.15.143:0/318984232 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb1e802fe70 msgr2=0x7fb1e8032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:54.225 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:54.224+0000 7fb207448700 1 --2- 172.21.15.143:0/318984232 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb1e802fe70 0x7fb1e8032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb1fc0096a0 tx=0x7fb1fc006b40).stop 2024-02-19T21:53:54.225 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:54.224+0000 7fb207448700 1 -- 172.21.15.143:0/318984232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 msgr2=0x7fb2000725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:54.225 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:54.224+0000 7fb207448700 1 --2- 172.21.15.143:0/318984232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000725e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb1f0002f40 tx=0x7fb1f0006ee0).stop 2024-02-19T21:53:54.225 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:54.224+0000 7fb207448700 1 -- 172.21.15.143:0/318984232 shutdown_connections 2024-02-19T21:53:54.225 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:54.224+0000 7fb207448700 1 --2- 172.21.15.143:0/318984232 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb1e802fe70 0x7fb1e8032330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:54.225 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:54.224+0000 7fb207448700 1 --2- 172.21.15.143:0/318984232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2000fbea0 0x7fb2000725e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:54.226 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:54.224+0000 7fb207448700 1 -- 172.21.15.143:0/318984232 >> 172.21.15.143:0/318984232 conn(0x7fb2000f76a0 msgr2=0x7fb2000f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:54.226 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:54.224+0000 7fb207448700 1 -- 172.21.15.143:0/318984232 shutdown_connections 2024-02-19T21:53:54.226 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:54.224+0000 7fb207448700 1 -- 172.21.15.143:0/318984232 wait complete. 2024-02-19T21:53:54.855 INFO:teuthology.orchestra.run.smithi143.stderr:Deploying mon service with default placement... 2024-02-19T21:53:54.855 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-19T21:53:55.379 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.374+0000 7f35cc2f6700 1 Processor -- start 2024-02-19T21:53:55.379 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.374+0000 7f35cc2f6700 1 -- start start 2024-02-19T21:53:55.379 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.374+0000 7f35cc2f6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c40feae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:55.379 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.374+0000 7f35cc2f6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c40fefd0 con 0x7f35c4070460 2024-02-19T21:53:55.379 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.374+0000 7f35ca092700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c40feae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:55.379 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.374+0000 7f35ca092700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c40feae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40774/0 (socket says 172.21.15.143:40774) 2024-02-19T21:53:55.379 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.374+0000 7f35ca092700 1 -- 172.21.15.143:0/356450197 learned_addr learned my addr 172.21.15.143:0/356450197 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:55.379 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.374+0000 7f35ca092700 1 -- 172.21.15.143:0/356450197 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c40ff110 con 0x7f35c4070460 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.374+0000 7f35ca092700 1 --2- 172.21.15.143:0/356450197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c40feae0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f35b400b5f0 tx=0x7f35b4006ee0).ready entity=mon.0 client_cookie=f7d1cdaa7ee7831f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.374+0000 7f35c9090700 1 -- 172.21.15.143:0/356450197 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b400d600 con 0x7f35c4070460 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.375+0000 7f35c9090700 1 -- 172.21.15.143:0/356450197 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f35b400dbf0 con 0x7f35c4070460 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.375+0000 7f35c9090700 1 -- 172.21.15.143:0/356450197 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b4011ce0 con 0x7f35c4070460 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.375+0000 7f35cc2f6700 1 -- 172.21.15.143:0/356450197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 msgr2=0x7f35c40feae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.375+0000 7f35cc2f6700 1 --2- 172.21.15.143:0/356450197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c40feae0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f35b400b5f0 tx=0x7f35b4006ee0).stop 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.375+0000 7f35cc2f6700 1 -- 172.21.15.143:0/356450197 shutdown_connections 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.375+0000 7f35cc2f6700 1 --2- 172.21.15.143:0/356450197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c40feae0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.375+0000 7f35cc2f6700 1 -- 172.21.15.143:0/356450197 >> 172.21.15.143:0/356450197 conn(0x7f35c40f6560 msgr2=0x7f35c40f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.375+0000 7f35cc2f6700 1 -- 172.21.15.143:0/356450197 shutdown_connections 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.375+0000 7f35cc2f6700 1 -- 172.21.15.143:0/356450197 wait complete. 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.376+0000 7f35cc2f6700 1 Processor -- start 2024-02-19T21:53:55.380 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.376+0000 7f35cc2f6700 1 -- start start 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.376+0000 7f35cc2f6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.376+0000 7f35cc2f6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c410b080 con 0x7f35c4070460 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.376+0000 7f35ca092700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.376+0000 7f35ca092700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c410ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40786/0 (socket says 172.21.15.143:40786) 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.376+0000 7f35ca092700 1 -- 172.21.15.143:0/3043061927 learned_addr learned my addr 172.21.15.143:0/3043061927 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.376+0000 7f35ca092700 1 -- 172.21.15.143:0/3043061927 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b400b200 con 0x7f35c4070460 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.376+0000 7f35ca092700 1 --2- 172.21.15.143:0/3043061927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c410ab90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f35b40115c0 tx=0x7f35b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.377+0000 7f35b2ffd700 1 -- 172.21.15.143:0/3043061927 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b401b410 con 0x7f35c4070460 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.377+0000 7f35b2ffd700 1 -- 172.21.15.143:0/3043061927 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f35b401ba00 con 0x7f35c4070460 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.377+0000 7f35cc2f6700 1 -- 172.21.15.143:0/3043061927 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c410b280 con 0x7f35c4070460 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.377+0000 7f35b2ffd700 1 -- 172.21.15.143:0/3043061927 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b401aaa0 con 0x7f35c4070460 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.377+0000 7f35cc2f6700 1 -- 172.21.15.143:0/3043061927 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c410b710 con 0x7f35c4070460 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.377+0000 7f35b2ffd700 1 -- 172.21.15.143:0/3043061927 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f35b400f070 con 0x7f35c4070460 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.377+0000 7f35b2ffd700 1 --2- 172.21.15.143:0/3043061927 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f35ac02fe70 0x7f35ac032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:55.381 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.377+0000 7f35b2ffd700 1 -- 172.21.15.143:0/3043061927 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f35b4040fd0 con 0x7f35c4070460 2024-02-19T21:53:55.382 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.378+0000 7f35c9891700 1 --2- 172.21.15.143:0/3043061927 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f35ac02fe70 0x7f35ac032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:55.382 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.378+0000 7f35cc2f6700 1 -- 172.21.15.143:0/3043061927 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c4045ad0 con 0x7f35c4070460 2024-02-19T21:53:55.382 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.378+0000 7f35c9891700 1 --2- 172.21.15.143:0/3043061927 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f35ac02fe70 0x7f35ac032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f35c00096a0 tx=0x7f35c0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:55.382 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.382+0000 7f35b2ffd700 1 -- 172.21.15.143:0/3043061927 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35b4016070 con 0x7f35c4070460 2024-02-19T21:53:55.556 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.555+0000 7f35cc2f6700 1 -- 172.21.15.143:0/3043061927 --> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f35c4104bf0 con 0x7f35ac02fe70 2024-02-19T21:53:55.561 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.561+0000 7f35b2ffd700 1 -- 172.21.15.143:0/3043061927 <== mgr.14114 v2:172.21.15.143:6800/1142990465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f35c4104bf0 con 0x7f35ac02fe70 2024-02-19T21:53:55.561 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-19T21:53:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.563+0000 7f35cc2f6700 1 -- 172.21.15.143:0/3043061927 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f35ac02fe70 msgr2=0x7f35ac032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.563+0000 7f35cc2f6700 1 --2- 172.21.15.143:0/3043061927 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f35ac02fe70 0x7f35ac032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f35c00096a0 tx=0x7f35c0006b40).stop 2024-02-19T21:53:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.563+0000 7f35cc2f6700 1 -- 172.21.15.143:0/3043061927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 msgr2=0x7f35c410ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.563+0000 7f35cc2f6700 1 --2- 172.21.15.143:0/3043061927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c410ab90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f35b40115c0 tx=0x7f35b4006ee0).stop 2024-02-19T21:53:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.563+0000 7f35cc2f6700 1 -- 172.21.15.143:0/3043061927 shutdown_connections 2024-02-19T21:53:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.563+0000 7f35cc2f6700 1 --2- 172.21.15.143:0/3043061927 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f35ac02fe70 0x7f35ac032330 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:55.563 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.563+0000 7f35cc2f6700 1 --2- 172.21.15.143:0/3043061927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35c4070460 0x7f35c410ab90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:55.564 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.563+0000 7f35cc2f6700 1 -- 172.21.15.143:0/3043061927 >> 172.21.15.143:0/3043061927 conn(0x7f35c40f6560 msgr2=0x7f35c40f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:55.564 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.563+0000 7f35cc2f6700 1 -- 172.21.15.143:0/3043061927 shutdown_connections 2024-02-19T21:53:55.564 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:55.563+0000 7f35cc2f6700 1 -- 172.21.15.143:0/3043061927 wait complete. 2024-02-19T21:53:55.582 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:55 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:53.622261+0000 mgr.smithi143.bissbc (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi143", "addr": "172.21.15.143", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:53:55.582 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:55 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:54.222534+0000 mon.smithi143 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:55.582 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:55 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:53:54.222820+0000 mgr.smithi143.bissbc (mgr.14114) 9 : cephadm [INF] Added host smithi143 2024-02-19T21:53:55.582 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:55 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:54.223035+0000 mon.smithi143 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:53:56.187 INFO:teuthology.orchestra.run.smithi143.stderr:Deploying mgr service with default placement... 2024-02-19T21:53:56.187 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-19T21:53:56.702 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.699+0000 7fb64853a700 1 Processor -- start 2024-02-19T21:53:56.702 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.699+0000 7fb64853a700 1 -- start start 2024-02-19T21:53:56.702 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.699+0000 7fb64853a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:56.702 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.699+0000 7fb64853a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6400fef40 con 0x7fb640070430 2024-02-19T21:53:56.702 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.699+0000 7fb6462d6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:56.702 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.700+0000 7fb6462d6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40802/0 (socket says 172.21.15.143:40802) 2024-02-19T21:53:56.702 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.700+0000 7fb6462d6700 1 -- 172.21.15.143:0/3470689273 learned_addr learned my addr 172.21.15.143:0/3470689273 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:56.702 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.700+0000 7fb6462d6700 1 -- 172.21.15.143:0/3470689273 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6400ff080 con 0x7fb640070430 2024-02-19T21:53:56.702 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.700+0000 7fb6462d6700 1 --2- 172.21.15.143:0/3470689273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fea50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb63000b5f0 tx=0x7fb630006ee0).ready entity=mon.0 client_cookie=bf3d9a21cdb0c52f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.700+0000 7fb6452d4700 1 -- 172.21.15.143:0/3470689273 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb63000d600 con 0x7fb640070430 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.700+0000 7fb6452d4700 1 -- 172.21.15.143:0/3470689273 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb63000dbf0 con 0x7fb640070430 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.700+0000 7fb6452d4700 1 -- 172.21.15.143:0/3470689273 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb630011cb0 con 0x7fb640070430 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.701+0000 7fb64853a700 1 -- 172.21.15.143:0/3470689273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 msgr2=0x7fb6400fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.701+0000 7fb64853a700 1 --2- 172.21.15.143:0/3470689273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fea50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb63000b5f0 tx=0x7fb630006ee0).stop 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.701+0000 7fb64853a700 1 -- 172.21.15.143:0/3470689273 shutdown_connections 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.701+0000 7fb64853a700 1 --2- 172.21.15.143:0/3470689273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fea50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.701+0000 7fb64853a700 1 -- 172.21.15.143:0/3470689273 >> 172.21.15.143:0/3470689273 conn(0x7fb6400f6530 msgr2=0x7fb6400f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.701+0000 7fb64853a700 1 -- 172.21.15.143:0/3470689273 shutdown_connections 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.701+0000 7fb64853a700 1 -- 172.21.15.143:0/3470689273 wait complete. 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.701+0000 7fb64853a700 1 Processor -- start 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.701+0000 7fb64853a700 1 -- start start 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.701+0000 7fb64853a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.701+0000 7fb64853a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6400fa950 con 0x7fb640070430 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.702+0000 7fb6462d6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:56.703 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.702+0000 7fb6462d6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40812/0 (socket says 172.21.15.143:40812) 2024-02-19T21:53:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.702+0000 7fb6462d6700 1 -- 172.21.15.143:0/2832891834 learned_addr learned my addr 172.21.15.143:0/2832891834 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.702+0000 7fb6462d6700 1 -- 172.21.15.143:0/2832891834 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63000b200 con 0x7fb640070430 2024-02-19T21:53:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.702+0000 7fb6462d6700 1 --2- 172.21.15.143:0/2832891834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fc4d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb630006800 tx=0x7fb630006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.702+0000 7fb62effd700 1 -- 172.21.15.143:0/2832891834 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb63001b570 con 0x7fb640070430 2024-02-19T21:53:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.702+0000 7fb62effd700 1 -- 172.21.15.143:0/2832891834 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb63001bb60 con 0x7fb640070430 2024-02-19T21:53:56.704 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.702+0000 7fb64853a700 1 -- 172.21.15.143:0/2832891834 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6400faaf0 con 0x7fb640070430 2024-02-19T21:53:56.705 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.702+0000 7fb62effd700 1 -- 172.21.15.143:0/2832891834 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb63001a9f0 con 0x7fb640070430 2024-02-19T21:53:56.705 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.702+0000 7fb64853a700 1 -- 172.21.15.143:0/2832891834 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6400faf80 con 0x7fb640070430 2024-02-19T21:53:56.705 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.703+0000 7fb64853a700 1 -- 172.21.15.143:0/2832891834 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb640045ad0 con 0x7fb640070430 2024-02-19T21:53:56.705 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.703+0000 7fb62effd700 1 -- 172.21.15.143:0/2832891834 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb63000f070 con 0x7fb640070430 2024-02-19T21:53:56.705 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.703+0000 7fb62effd700 1 --2- 172.21.15.143:0/2832891834 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb62802fe10 0x7fb6280322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:56.705 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.703+0000 7fb62effd700 1 -- 172.21.15.143:0/2832891834 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb630041440 con 0x7fb640070430 2024-02-19T21:53:56.705 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.703+0000 7fb645ad5700 1 --2- 172.21.15.143:0/2832891834 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb62802fe10 0x7fb6280322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:56.705 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.704+0000 7fb645ad5700 1 --2- 172.21.15.143:0/2832891834 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb62802fe10 0x7fb6280322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb63c0096a0 tx=0x7fb63c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:56.707 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.708+0000 7fb62effd700 1 -- 172.21.15.143:0/2832891834 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6300226d0 con 0x7fb640070430 2024-02-19T21:53:56.881 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.881+0000 7fb64853a700 1 -- 172.21.15.143:0/2832891834 --> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6400fff40 con 0x7fb62802fe10 2024-02-19T21:53:56.885 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.886+0000 7fb62effd700 1 -- 172.21.15.143:0/2832891834 <== mgr.14114 v2:172.21.15.143:6800/1142990465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb6400fff40 con 0x7fb62802fe10 2024-02-19T21:53:56.886 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-19T21:53:56.888 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.887+0000 7fb64853a700 1 -- 172.21.15.143:0/2832891834 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb62802fe10 msgr2=0x7fb6280322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:56.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.887+0000 7fb64853a700 1 --2- 172.21.15.143:0/2832891834 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb62802fe10 0x7fb6280322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb63c0096a0 tx=0x7fb63c006b40).stop 2024-02-19T21:53:56.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.887+0000 7fb64853a700 1 -- 172.21.15.143:0/2832891834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 msgr2=0x7fb6400fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:56.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.887+0000 7fb64853a700 1 --2- 172.21.15.143:0/2832891834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fc4d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb630006800 tx=0x7fb630006ee0).stop 2024-02-19T21:53:56.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.887+0000 7fb64853a700 1 -- 172.21.15.143:0/2832891834 shutdown_connections 2024-02-19T21:53:56.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.887+0000 7fb64853a700 1 --2- 172.21.15.143:0/2832891834 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fb62802fe10 0x7fb6280322d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:56.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.887+0000 7fb64853a700 1 --2- 172.21.15.143:0/2832891834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb640070430 0x7fb6400fc4d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:56.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.887+0000 7fb64853a700 1 -- 172.21.15.143:0/2832891834 >> 172.21.15.143:0/2832891834 conn(0x7fb6400f6530 msgr2=0x7fb6400f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:56.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.888+0000 7fb64853a700 1 -- 172.21.15.143:0/2832891834 shutdown_connections 2024-02-19T21:53:56.889 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:56.888+0000 7fb64853a700 1 -- 172.21.15.143:0/2832891834 wait complete. 2024-02-19T21:53:56.907 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:55.558136+0000 mgr.smithi143.bissbc (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:53:56.907 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:53:55.558900+0000 mgr.smithi143.bissbc (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-19T21:53:56.907 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:55.561916+0000 mon.smithi143 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:57.517 INFO:teuthology.orchestra.run.smithi143.stderr:Deploying crash service with default placement... 2024-02-19T21:53:57.517 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-19T21:53:58.021 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.015+0000 7f5830aed700 1 Processor -- start 2024-02-19T21:53:58.021 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.015+0000 7f5830aed700 1 -- start start 2024-02-19T21:53:58.021 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.016+0000 7f5830aed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:58.022 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.016+0000 7f5830aed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c0fc8c0 con 0x7f582c0703b0 2024-02-19T21:53:58.022 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.016+0000 7f582a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:58.022 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.016+0000 7f582a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40826/0 (socket says 172.21.15.143:40826) 2024-02-19T21:53:58.022 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.016+0000 7f582a59c700 1 -- 172.21.15.143:0/3113221356 learned_addr learned my addr 172.21.15.143:0/3113221356 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:58.022 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.016+0000 7f582a59c700 1 -- 172.21.15.143:0/3113221356 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c1013c0 con 0x7f582c0703b0 2024-02-19T21:53:58.022 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.016+0000 7f582a59c700 1 --2- 172.21.15.143:0/3113221356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c100ed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f581400b5f0 tx=0x7f5814006ee0).ready entity=mon.0 client_cookie=e7dc199a26bcf47b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:58.022 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.016+0000 7f582959a700 1 -- 172.21.15.143:0/3113221356 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f581400d600 con 0x7f582c0703b0 2024-02-19T21:53:58.022 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.016+0000 7f582959a700 1 -- 172.21.15.143:0/3113221356 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f581400dbf0 con 0x7f582c0703b0 2024-02-19T21:53:58.022 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.016+0000 7f582959a700 1 -- 172.21.15.143:0/3113221356 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5814011cb0 con 0x7f582c0703b0 2024-02-19T21:53:58.022 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.017+0000 7f5830aed700 1 -- 172.21.15.143:0/3113221356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 msgr2=0x7f582c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.017+0000 7f5830aed700 1 --2- 172.21.15.143:0/3113221356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c100ed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f581400b5f0 tx=0x7f5814006ee0).stop 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.017+0000 7f5830aed700 1 -- 172.21.15.143:0/3113221356 shutdown_connections 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.017+0000 7f5830aed700 1 --2- 172.21.15.143:0/3113221356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c100ed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.017+0000 7f5830aed700 1 -- 172.21.15.143:0/3113221356 >> 172.21.15.143:0/3113221356 conn(0x7f582c0f6530 msgr2=0x7f582c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.017+0000 7f5830aed700 1 -- 172.21.15.143:0/3113221356 shutdown_connections 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.017+0000 7f5830aed700 1 -- 172.21.15.143:0/3113221356 wait complete. 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.017+0000 7f5830aed700 1 Processor -- start 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.017+0000 7f5830aed700 1 -- start start 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f5830aed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c10cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f5830aed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c10d2c0 con 0x7f582c0703b0 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f582a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c10cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f582a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c10cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:40834/0 (socket says 172.21.15.143:40834) 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f582a59c700 1 -- 172.21.15.143:0/3741811916 learned_addr learned my addr 172.21.15.143:0/3741811916 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:58.023 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f582a59c700 1 -- 172.21.15.143:0/3741811916 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581400b200 con 0x7f582c0703b0 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f582a59c700 1 --2- 172.21.15.143:0/3741811916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c10cdd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f581400df90 tx=0x7f5814006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f5822ffd700 1 -- 172.21.15.143:0/3741811916 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f581401b410 con 0x7f582c0703b0 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f5822ffd700 1 -- 172.21.15.143:0/3741811916 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f581401ba00 con 0x7f582c0703b0 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f5830aed700 1 -- 172.21.15.143:0/3741811916 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582c10d4c0 con 0x7f582c0703b0 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f5822ffd700 1 -- 172.21.15.143:0/3741811916 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f581401aa30 con 0x7f582c0703b0 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.018+0000 7f5830aed700 1 -- 172.21.15.143:0/3741811916 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582c10e100 con 0x7f582c0703b0 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.019+0000 7f5830aed700 1 -- 172.21.15.143:0/3741811916 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582c10d950 con 0x7f582c0703b0 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.020+0000 7f5822ffd700 1 -- 172.21.15.143:0/3741811916 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f581400f070 con 0x7f582c0703b0 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.020+0000 7f5822ffd700 1 --2- 172.21.15.143:0/3741811916 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f580c02fe10 0x7f580c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.020+0000 7f5822ffd700 1 -- 172.21.15.143:0/3741811916 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f58140418c0 con 0x7f582c0703b0 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.020+0000 7f5829d9b700 1 --2- 172.21.15.143:0/3741811916 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f580c02fe10 0x7f580c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:58.024 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.020+0000 7f5829d9b700 1 --2- 172.21.15.143:0/3741811916 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f580c02fe10 0x7f580c0322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f581c0096a0 tx=0x7f581c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:58.025 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.024+0000 7f5822ffd700 1 -- 172.21.15.143:0/3741811916 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5814019af0 con 0x7f582c0703b0 2024-02-19T21:53:58.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:57 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:56.883133+0000 mgr.smithi143.bissbc (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:53:58.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:57 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:53:56.883653+0000 mgr.smithi143.bissbc (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-19T21:53:58.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:57 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:56.886506+0000 mon.smithi143 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' 2024-02-19T21:53:58.197 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.198+0000 7f5830aed700 1 -- 172.21.15.143:0/3741811916 --> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f582c10e340 con 0x7f580c02fe10 2024-02-19T21:53:58.201 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.202+0000 7f5822ffd700 1 -- 172.21.15.143:0/3741811916 <== mgr.14114 v2:172.21.15.143:6800/1142990465 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f582c10e340 con 0x7f580c02fe10 2024-02-19T21:53:58.201 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-19T21:53:58.204 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.204+0000 7f5830aed700 1 -- 172.21.15.143:0/3741811916 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f580c02fe10 msgr2=0x7f580c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:58.204 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.204+0000 7f5830aed700 1 --2- 172.21.15.143:0/3741811916 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f580c02fe10 0x7f580c0322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f581c0096a0 tx=0x7f581c006b40).stop 2024-02-19T21:53:58.204 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.204+0000 7f5830aed700 1 -- 172.21.15.143:0/3741811916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 msgr2=0x7f582c10cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:58.204 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.204+0000 7f5830aed700 1 --2- 172.21.15.143:0/3741811916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c10cdd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f581400df90 tx=0x7f5814006ee0).stop 2024-02-19T21:53:58.204 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.204+0000 7f5830aed700 1 -- 172.21.15.143:0/3741811916 shutdown_connections 2024-02-19T21:53:58.204 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.204+0000 7f5830aed700 1 --2- 172.21.15.143:0/3741811916 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7f580c02fe10 0x7f580c0322d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:58.204 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.204+0000 7f5830aed700 1 --2- 172.21.15.143:0/3741811916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f582c0703b0 0x7f582c10cdd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:58.204 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.204+0000 7f5830aed700 1 -- 172.21.15.143:0/3741811916 >> 172.21.15.143:0/3741811916 conn(0x7f582c0f6530 msgr2=0x7f582c0f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:58.204 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.204+0000 7f5830aed700 1 -- 172.21.15.143:0/3741811916 shutdown_connections 2024-02-19T21:53:58.205 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:58.204+0000 7f5830aed700 1 -- 172.21.15.143:0/3741811916 wait complete. 2024-02-19T21:53:58.816 INFO:teuthology.orchestra.run.smithi143.stderr:Enabling mgr prometheus module... 2024-02-19T21:53:58.816 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-19T21:53:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.314+0000 7efd12e39700 1 Processor -- start 2024-02-19T21:53:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.314+0000 7efd12e39700 1 -- start start 2024-02-19T21:53:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.315+0000 7efd12e39700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.315+0000 7efd12e39700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c0fef80 con 0x7efd0c0703f0 2024-02-19T21:53:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.315+0000 7efd10bd5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.315+0000 7efd10bd5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35202/0 (socket says 172.21.15.143:35202) 2024-02-19T21:53:59.317 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.315+0000 7efd10bd5700 1 -- 172.21.15.143:0/2311352287 learned_addr learned my addr 172.21.15.143:0/2311352287 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.315+0000 7efd10bd5700 1 -- 172.21.15.143:0/2311352287 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c0ff0c0 con 0x7efd0c0703f0 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.315+0000 7efd10bd5700 1 --2- 172.21.15.143:0/2311352287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fea90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7efd0000b5f0 tx=0x7efd00006ee0).ready entity=mon.0 client_cookie=375a2a9ca3df6476 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.315+0000 7efd0b7fe700 1 -- 172.21.15.143:0/2311352287 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd0000d600 con 0x7efd0c0703f0 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.316+0000 7efd0b7fe700 1 -- 172.21.15.143:0/2311352287 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efd0000dbf0 con 0x7efd0c0703f0 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.316+0000 7efd0b7fe700 1 -- 172.21.15.143:0/2311352287 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd00011ce0 con 0x7efd0c0703f0 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.316+0000 7efd12e39700 1 -- 172.21.15.143:0/2311352287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 msgr2=0x7efd0c0fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.316+0000 7efd12e39700 1 --2- 172.21.15.143:0/2311352287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fea90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7efd0000b5f0 tx=0x7efd00006ee0).stop 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.316+0000 7efd12e39700 1 -- 172.21.15.143:0/2311352287 shutdown_connections 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.316+0000 7efd12e39700 1 --2- 172.21.15.143:0/2311352287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fea90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.316+0000 7efd12e39700 1 -- 172.21.15.143:0/2311352287 >> 172.21.15.143:0/2311352287 conn(0x7efd0c0f6530 msgr2=0x7efd0c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.316+0000 7efd12e39700 1 -- 172.21.15.143:0/2311352287 shutdown_connections 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.316+0000 7efd12e39700 1 -- 172.21.15.143:0/2311352287 wait complete. 2024-02-19T21:53:59.318 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.317+0000 7efd12e39700 1 Processor -- start 2024-02-19T21:53:59.319 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.317+0000 7efd12e39700 1 -- start start 2024-02-19T21:53:59.319 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.317+0000 7efd12e39700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:59.319 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.317+0000 7efd12e39700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c0fa990 con 0x7efd0c0703f0 2024-02-19T21:53:59.319 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.317+0000 7efd10bd5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:59.319 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.317+0000 7efd10bd5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35206/0 (socket says 172.21.15.143:35206) 2024-02-19T21:53:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.317+0000 7efd10bd5700 1 -- 172.21.15.143:0/2701633097 learned_addr learned my addr 172.21.15.143:0/2701633097 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:53:59.320 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.318+0000 7efd10bd5700 1 -- 172.21.15.143:0/2701633097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0000b200 con 0x7efd0c0703f0 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.318+0000 7efd10bd5700 1 --2- 172.21.15.143:0/2701633097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fc510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7efd00011ed0 tx=0x7efd00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.318+0000 7efd097fa700 1 -- 172.21.15.143:0/2701633097 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd0001b570 con 0x7efd0c0703f0 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.318+0000 7efd097fa700 1 -- 172.21.15.143:0/2701633097 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efd0001bb60 con 0x7efd0c0703f0 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.318+0000 7efd12e39700 1 -- 172.21.15.143:0/2701633097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c0fab30 con 0x7efd0c0703f0 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.318+0000 7efd097fa700 1 -- 172.21.15.143:0/2701633097 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd0001aad0 con 0x7efd0c0703f0 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.318+0000 7efd12e39700 1 -- 172.21.15.143:0/2701633097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c0fafc0 con 0x7efd0c0703f0 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.319+0000 7efd097fa700 1 -- 172.21.15.143:0/2701633097 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7efd0000f070 con 0x7efd0c0703f0 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.319+0000 7efd097fa700 1 --2- 172.21.15.143:0/2701633097 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7efcf002fe10 0x7efcf00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.319+0000 7efd097fa700 1 -- 172.21.15.143:0/2701633097 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7efd00042260 con 0x7efd0c0703f0 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.319+0000 7efd0bfff700 1 --2- 172.21.15.143:0/2701633097 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7efcf002fe10 0x7efcf00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.319+0000 7efd12e39700 1 -- 172.21.15.143:0/2701633097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0c059100 con 0x7efd0c0703f0 2024-02-19T21:53:59.321 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.319+0000 7efd0bfff700 1 --2- 172.21.15.143:0/2701633097 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7efcf002fe10 0x7efcf00322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7efcf80096a0 tx=0x7efcf8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:53:59.323 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.323+0000 7efd097fa700 1 -- 172.21.15.143:0/2701633097 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd00016070 con 0x7efd0c0703f0 2024-02-19T21:53:59.507 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:53:59.508+0000 7efd12e39700 1 -- 172.21.15.143:0/2701633097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7efd0c045ad0 con 0x7efd0c0703f0 2024-02-19T21:53:59.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:58.199745+0000 mgr.smithi143.bissbc (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:53:59.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:53:58.200387+0000 mgr.smithi143.bissbc (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-19T21:53:59.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:53:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:58.202540+0000 mon.smithi143 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.143:0/3772217404' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:00.208 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.205+0000 7efd097fa700 1 -- 172.21.15.143:0/2701633097 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7efd00024ca0 con 0x7efd0c0703f0 2024-02-19T21:54:00.208 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.206+0000 7efd097fa700 1 -- 172.21.15.143:0/2701633097 <== mon.0 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7efd00019d50 con 0x7efd0c0703f0 2024-02-19T21:54:00.208 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.208+0000 7efd12e39700 1 -- 172.21.15.143:0/2701633097 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7efcf002fe10 msgr2=0x7efcf00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:00.208 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.208+0000 7efd12e39700 1 --2- 172.21.15.143:0/2701633097 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7efcf002fe10 0x7efcf00322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7efcf80096a0 tx=0x7efcf8006b40).stop 2024-02-19T21:54:00.209 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.208+0000 7efd12e39700 1 -- 172.21.15.143:0/2701633097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 msgr2=0x7efd0c0fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:00.210 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.208+0000 7efd12e39700 1 --2- 172.21.15.143:0/2701633097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fc510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7efd00011ed0 tx=0x7efd00006ee0).stop 2024-02-19T21:54:00.210 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.209+0000 7efd12e39700 1 -- 172.21.15.143:0/2701633097 shutdown_connections 2024-02-19T21:54:00.210 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.209+0000 7efd12e39700 1 --2- 172.21.15.143:0/2701633097 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7efcf002fe10 0x7efcf00322d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:00.210 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.209+0000 7efd12e39700 1 --2- 172.21.15.143:0/2701633097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd0c0703f0 0x7efd0c0fc510 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:00.210 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.209+0000 7efd12e39700 1 -- 172.21.15.143:0/2701633097 >> 172.21.15.143:0/2701633097 conn(0x7efd0c0f6530 msgr2=0x7efd0c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:00.210 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.209+0000 7efd12e39700 1 -- 172.21.15.143:0/2701633097 shutdown_connections 2024-02-19T21:54:00.210 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:00.209+0000 7efd12e39700 1 -- 172.21.15.143:0/2701633097 wait complete. 2024-02-19T21:54:00.543 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:53:59.509079+0000 mon.smithi143 (mon.0) 59 : audit [INF] from='client.? 172.21.15.143:0/2701633097' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T21:54:00.838 INFO:teuthology.orchestra.run.smithi143.stderr:Deploying prometheus service with default placement... 2024-02-19T21:54:00.839 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-19T21:54:01.384 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.375+0000 7fba96d43700 1 Processor -- start 2024-02-19T21:54:01.384 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.375+0000 7fba96d43700 1 -- start start 2024-02-19T21:54:01.384 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.375+0000 7fba96d43700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba900fd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:01.384 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.375+0000 7fba96d43700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba900fddd0 con 0x7fba900fd4c0 2024-02-19T21:54:01.384 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.375+0000 7fba94adf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba900fd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:01.384 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.375+0000 7fba94adf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba900fd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35240/0 (socket says 172.21.15.143:35240) 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.375+0000 7fba94adf700 1 -- 172.21.15.143:0/1270338869 learned_addr learned my addr 172.21.15.143:0/1270338869 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.375+0000 7fba94adf700 1 -- 172.21.15.143:0/1270338869 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba900fdf10 con 0x7fba900fd4c0 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.376+0000 7fba94adf700 1 --2- 172.21.15.143:0/1270338869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba900fd8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fba8400b5f0 tx=0x7fba84006ee0).ready entity=mon.0 client_cookie=bb6744e8041f707b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.376+0000 7fba8f7fe700 1 -- 172.21.15.143:0/1270338869 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba8400d600 con 0x7fba900fd4c0 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.376+0000 7fba8f7fe700 1 -- 172.21.15.143:0/1270338869 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fba8400dbf0 con 0x7fba900fd4c0 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.376+0000 7fba8f7fe700 1 -- 172.21.15.143:0/1270338869 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba84011cb0 con 0x7fba900fd4c0 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.376+0000 7fba96d43700 1 -- 172.21.15.143:0/1270338869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 msgr2=0x7fba900fd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.376+0000 7fba96d43700 1 --2- 172.21.15.143:0/1270338869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba900fd8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fba8400b5f0 tx=0x7fba84006ee0).stop 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.376+0000 7fba96d43700 1 -- 172.21.15.143:0/1270338869 shutdown_connections 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.376+0000 7fba96d43700 1 --2- 172.21.15.143:0/1270338869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba900fd8e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.376+0000 7fba96d43700 1 -- 172.21.15.143:0/1270338869 >> 172.21.15.143:0/1270338869 conn(0x7fba900f5d20 msgr2=0x7fba900f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.376+0000 7fba96d43700 1 -- 172.21.15.143:0/1270338869 shutdown_connections 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.376+0000 7fba96d43700 1 -- 172.21.15.143:0/1270338869 wait complete. 2024-02-19T21:54:01.385 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.377+0000 7fba96d43700 1 Processor -- start 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.377+0000 7fba96d43700 1 -- start start 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.377+0000 7fba96d43700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba9010ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.377+0000 7fba96d43700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9010b030 con 0x7fba900fd4c0 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.378+0000 7fba94adf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba9010ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.378+0000 7fba94adf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba9010ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35254/0 (socket says 172.21.15.143:35254) 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.378+0000 7fba94adf700 1 -- 172.21.15.143:0/4024356814 learned_addr learned my addr 172.21.15.143:0/4024356814 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.378+0000 7fba94adf700 1 -- 172.21.15.143:0/4024356814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8400b200 con 0x7fba900fd4c0 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.378+0000 7fba94adf700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba9010ab40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fba8400bf40 tx=0x7fba84006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.378+0000 7fba8d7fa700 1 -- 172.21.15.143:0/4024356814 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba8401b570 con 0x7fba900fd4c0 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.378+0000 7fba8d7fa700 1 -- 172.21.15.143:0/4024356814 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fba8401bb60 con 0x7fba900fd4c0 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.378+0000 7fba96d43700 1 -- 172.21.15.143:0/4024356814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9010b230 con 0x7fba900fd4c0 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.378+0000 7fba8d7fa700 1 -- 172.21.15.143:0/4024356814 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba8401aa20 con 0x7fba900fd4c0 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.378+0000 7fba96d43700 1 -- 172.21.15.143:0/4024356814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9010b6c0 con 0x7fba900fd4c0 2024-02-19T21:54:01.386 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.379+0000 7fba8d7fa700 1 -- 172.21.15.143:0/4024356814 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fba8400f070 con 0x7fba900fd4c0 2024-02-19T21:54:01.387 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.379+0000 7fba8d7fa700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 0x7fba74032380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:01.387 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.379+0000 7fba8d7fa700 1 -- 172.21.15.143:0/4024356814 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fba84041450 con 0x7fba900fd4c0 2024-02-19T21:54:01.387 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.379+0000 7fba8ffff700 1 -- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 msgr2=0x7fba74032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1142990465 2024-02-19T21:54:01.387 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.379+0000 7fba8ffff700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 0x7fba74032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T21:54:01.387 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.379+0000 7fba96d43700 1 -- 172.21.15.143:0/4024356814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba90045ad0 con 0x7fba900fd4c0 2024-02-19T21:54:01.387 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.386+0000 7fba8d7fa700 1 -- 172.21.15.143:0/4024356814 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba84016070 con 0x7fba900fd4c0 2024-02-19T21:54:01.579 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.580+0000 7fba8ffff700 1 -- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 msgr2=0x7fba74032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1142990465 2024-02-19T21:54:01.580 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.580+0000 7fba8ffff700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 0x7fba74032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T21:54:01.617 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.617+0000 7fba96d43700 1 -- 172.21.15.143:0/4024356814 --> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fba901046f0 con 0x7fba7402fec0 2024-02-19T21:54:01.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:01 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:00.207008+0000 mon.smithi143 (mon.0) 60 : audit [INF] from='client.? 172.21.15.143:0/2701633097' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T21:54:01.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:01 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:00.207066+0000 mon.smithi143 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi143.bissbc(active, since 20s) 2024-02-19T21:54:01.981 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.981+0000 7fba8ffff700 1 -- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 msgr2=0x7fba74032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1142990465 2024-02-19T21:54:01.981 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:01.981+0000 7fba8ffff700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 0x7fba74032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T21:54:02.782 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:02.782+0000 7fba8ffff700 1 -- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 msgr2=0x7fba74032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1142990465 2024-02-19T21:54:02.782 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:02.782+0000 7fba8ffff700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 0x7fba74032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T21:54:04.384 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:04.383+0000 7fba8ffff700 1 -- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 msgr2=0x7fba74032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/1142990465 2024-02-19T21:54:04.384 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:04.383+0000 7fba8ffff700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 0x7fba74032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-19T21:54:05.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:05.248+0000 7fba8d7fa700 1 -- 172.21.15.143:0/4024356814 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fba8401bcd0 con 0x7fba900fd4c0 2024-02-19T21:54:05.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:05.248+0000 7fba8d7fa700 1 -- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 msgr2=0x7fba74032380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:54:05.249 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:05.248+0000 7fba8d7fa700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 0x7fba74032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:54:06.251 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.250+0000 7fba8d7fa700 1 -- 172.21.15.143:0/4024356814 <== mon.0 v2:172.21.15.143:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fba840432c0 con 0x7fba900fd4c0 2024-02-19T21:54:06.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.250+0000 7fba8d7fa700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fba74033ce0 0x7fba740360d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:06.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.250+0000 7fba8d7fa700 1 -- 172.21.15.143:0/4024356814 --> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fba901046f0 con 0x7fba74033ce0 2024-02-19T21:54:06.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.251+0000 7fba8ffff700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fba74033ce0 0x7fba740360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:06.252 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.251+0000 7fba8ffff700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fba74033ce0 0x7fba740360d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fba7c006fd0 tx=0x7fba7c006d30).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:06.255 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.255+0000 7fba8d7fa700 1 -- 172.21.15.143:0/4024356814 <== mgr.14142 v2:172.21.15.143:6800/766735871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fba901046f0 con 0x7fba74033ce0 2024-02-19T21:54:06.256 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-19T21:54:06.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.258+0000 7fba96d43700 1 -- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fba74033ce0 msgr2=0x7fba740360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:06.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.258+0000 7fba96d43700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fba74033ce0 0x7fba740360d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fba7c006fd0 tx=0x7fba7c006d30).stop 2024-02-19T21:54:06.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.258+0000 7fba96d43700 1 -- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 msgr2=0x7fba9010ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:06.258 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.258+0000 7fba96d43700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba9010ab40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fba8400bf40 tx=0x7fba84006ee0).stop 2024-02-19T21:54:06.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.258+0000 7fba96d43700 1 -- 172.21.15.143:0/4024356814 shutdown_connections 2024-02-19T21:54:06.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.258+0000 7fba96d43700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fba74033ce0 0x7fba740360d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:06.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.258+0000 7fba96d43700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba900fd4c0 0x7fba9010ab40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:06.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.258+0000 7fba96d43700 1 --2- 172.21.15.143:0/4024356814 >> [v2:172.21.15.143:6800/1142990465,v1:172.21.15.143:6801/1142990465] conn(0x7fba7402fec0 0x7fba74032380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:54:06.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.258+0000 7fba96d43700 1 -- 172.21.15.143:0/4024356814 >> 172.21.15.143:0/4024356814 conn(0x7fba900f5d20 msgr2=0x7fba900ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:06.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.258+0000 7fba96d43700 1 -- 172.21.15.143:0/4024356814 shutdown_connections 2024-02-19T21:54:06.259 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:06.258+0000 7fba96d43700 1 -- 172.21.15.143:0/4024356814 wait complete. 2024-02-19T21:54:06.643 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:05.196704+0000 mon.smithi143 (mon.0) 62 : cluster [INF] Active manager daemon smithi143.bissbc restarted 2024-02-19T21:54:06.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:05.197209+0000 mon.smithi143 (mon.0) 63 : cluster [INF] Activating manager daemon smithi143.bissbc 2024-02-19T21:54:06.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:05.199513+0000 mon.smithi143 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-19T21:54:06.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:05.250273+0000 mon.smithi143 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi143.bissbc(active, starting, since 0.0531838s) 2024-02-19T21:54:06.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:05.251307+0000 mon.smithi143 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-02-19T21:54:06.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:05.251915+0000 mon.smithi143 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mgr metadata", "who": "smithi143.bissbc", "id": "smithi143.bissbc"}]: dispatch 2024-02-19T21:54:06.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:05.252533+0000 mon.smithi143 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T21:54:06.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:05.252679+0000 mon.smithi143 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T21:54:06.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:05.252805+0000 mon.smithi143 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T21:54:06.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:05.266155+0000 mon.smithi143 (mon.0) 71 : cluster [INF] Manager daemon smithi143.bissbc is now available 2024-02-19T21:54:06.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:05.362400+0000 mon.smithi143 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:54:06.644 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:05.363878+0000 mon.smithi143 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:54:06.645 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:05.378322+0000 mon.smithi143 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:54:06.645 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:05.386720+0000 mon.smithi143 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T21:54:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:Deploying grafana service with default placement... 2024-02-19T21:54:06.879 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-19T21:54:07.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:07 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:06.253154+0000 mon.smithi143 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi143.bissbc(active, since 1.05606s) 2024-02-19T21:54:07.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:07 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:54:06.254280+0000 mgr.smithi143.bissbc (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-19T21:54:07.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:07 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:06.256339+0000 mon.smithi143 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:07.445 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.442+0000 7f0f5a3c8700 1 Processor -- start 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.442+0000 7f0f5a3c8700 1 -- start start 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.442+0000 7f0f5a3c8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f540fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.442+0000 7f0f5a3c8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f540fb6a0 con 0x7f0f540fad90 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.442+0000 7f0f53fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f540fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.443+0000 7f0f53fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f540fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35304/0 (socket says 172.21.15.143:35304) 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.443+0000 7f0f53fff700 1 -- 172.21.15.143:0/428382714 learned_addr learned my addr 172.21.15.143:0/428382714 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.443+0000 7f0f53fff700 1 -- 172.21.15.143:0/428382714 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f540fb7e0 con 0x7f0f540fad90 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.443+0000 7f0f53fff700 1 --2- 172.21.15.143:0/428382714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f540fb1b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f0f4800b5f0 tx=0x7f0f48006ee0).ready entity=mon.0 client_cookie=94c730e26f9a6d20 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.443+0000 7f0f52ffd700 1 -- 172.21.15.143:0/428382714 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f4800d600 con 0x7f0f540fad90 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.443+0000 7f0f52ffd700 1 -- 172.21.15.143:0/428382714 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0f4800dbf0 con 0x7f0f540fad90 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.443+0000 7f0f52ffd700 1 -- 172.21.15.143:0/428382714 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f48011cb0 con 0x7f0f540fad90 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.444+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/428382714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 msgr2=0x7f0f540fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:07.446 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.444+0000 7f0f5a3c8700 1 --2- 172.21.15.143:0/428382714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f540fb1b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f0f4800b5f0 tx=0x7f0f48006ee0).stop 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.444+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/428382714 shutdown_connections 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.444+0000 7f0f5a3c8700 1 --2- 172.21.15.143:0/428382714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f540fb1b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.444+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/428382714 >> 172.21.15.143:0/428382714 conn(0x7f0f540f6530 msgr2=0x7f0f540f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.444+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/428382714 shutdown_connections 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.444+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/428382714 wait complete. 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.444+0000 7f0f5a3c8700 1 Processor -- start 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.444+0000 7f0f5a3c8700 1 -- start start 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.444+0000 7f0f5a3c8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f54108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.445+0000 7f0f5a3c8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f54108e10 con 0x7f0f540fad90 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.445+0000 7f0f53fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f54108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.445+0000 7f0f53fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f54108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35310/0 (socket says 172.21.15.143:35310) 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.445+0000 7f0f53fff700 1 -- 172.21.15.143:0/2787438622 learned_addr learned my addr 172.21.15.143:0/2787438622 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.445+0000 7f0f53fff700 1 -- 172.21.15.143:0/2787438622 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4800b200 con 0x7f0f540fad90 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.445+0000 7f0f53fff700 1 --2- 172.21.15.143:0/2787438622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f54108920 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0f48011ea0 tx=0x7f0f48006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.445+0000 7f0f50ff9700 1 -- 172.21.15.143:0/2787438622 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f4801b570 con 0x7f0f540fad90 2024-02-19T21:54:07.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.445+0000 7f0f50ff9700 1 -- 172.21.15.143:0/2787438622 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0f4801bb60 con 0x7f0f540fad90 2024-02-19T21:54:07.448 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.445+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/2787438622 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f54109010 con 0x7f0f540fad90 2024-02-19T21:54:07.448 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.445+0000 7f0f50ff9700 1 -- 172.21.15.143:0/2787438622 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f4801a8d0 con 0x7f0f540fad90 2024-02-19T21:54:07.448 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.445+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/2787438622 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f541094a0 con 0x7f0f540fad90 2024-02-19T21:54:07.449 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.446+0000 7f0f50ff9700 1 -- 172.21.15.143:0/2787438622 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f0f4800f070 con 0x7f0f540fad90 2024-02-19T21:54:07.449 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.446+0000 7f0f50ff9700 1 --2- 172.21.15.143:0/2787438622 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f0f40032160 0x7f0f40034620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:07.449 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.446+0000 7f0f50ff9700 1 -- 172.21.15.143:0/2787438622 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0f480415d0 con 0x7f0f540fad90 2024-02-19T21:54:07.449 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.446+0000 7f0f537fe700 1 --2- 172.21.15.143:0/2787438622 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f0f40032160 0x7f0f40034620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:07.449 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.446+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/2787438622 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f54045ad0 con 0x7f0f540fad90 2024-02-19T21:54:07.449 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.447+0000 7f0f537fe700 1 --2- 172.21.15.143:0/2787438622 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f0f40032160 0x7f0f40034620 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0f440096a0 tx=0x7f0f44006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:07.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.451+0000 7f0f50ff9700 1 -- 172.21.15.143:0/2787438622 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f48016080 con 0x7f0f540fad90 2024-02-19T21:54:07.626 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.626+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/2787438622 --> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f541027b0 con 0x7f0f40032160 2024-02-19T21:54:07.632 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.632+0000 7f0f50ff9700 1 -- 172.21.15.143:0/2787438622 <== mgr.14142 v2:172.21.15.143:6800/766735871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f0f541027b0 con 0x7f0f40032160 2024-02-19T21:54:07.632 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-19T21:54:07.635 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.634+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/2787438622 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f0f40032160 msgr2=0x7f0f40034620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:07.635 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.634+0000 7f0f5a3c8700 1 --2- 172.21.15.143:0/2787438622 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f0f40032160 0x7f0f40034620 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0f440096a0 tx=0x7f0f44006b40).stop 2024-02-19T21:54:07.635 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.634+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/2787438622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 msgr2=0x7f0f54108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:07.635 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.634+0000 7f0f5a3c8700 1 --2- 172.21.15.143:0/2787438622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f54108920 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0f48011ea0 tx=0x7f0f48006ee0).stop 2024-02-19T21:54:07.635 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.634+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/2787438622 shutdown_connections 2024-02-19T21:54:07.635 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.635+0000 7f0f5a3c8700 1 --2- 172.21.15.143:0/2787438622 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f0f40032160 0x7f0f40034620 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:07.635 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.635+0000 7f0f5a3c8700 1 --2- 172.21.15.143:0/2787438622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f540fad90 0x7f0f54108920 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:07.635 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.635+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/2787438622 >> 172.21.15.143:0/2787438622 conn(0x7f0f540f6530 msgr2=0x7f0f540ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:07.635 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.635+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/2787438622 shutdown_connections 2024-02-19T21:54:07.635 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:07.635+0000 7f0f5a3c8700 1 -- 172.21.15.143:0/2787438622 wait complete. 2024-02-19T21:54:08.260 INFO:teuthology.orchestra.run.smithi143.stderr:Deploying node-exporter service with default placement... 2024-02-19T21:54:08.260 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-19T21:54:08.515 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:07.257407+0000 mon.smithi143 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi143.bissbc(active, since 2s) 2024-02-19T21:54:08.515 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:07.627859+0000 mgr.smithi143.bissbc (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:54:08.516 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:54:07.628384+0000 mgr.smithi143.bissbc (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-19T21:54:08.516 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:07.633344+0000 mon.smithi143 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.766+0000 7f1939176700 1 Processor -- start 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.766+0000 7f1939176700 1 -- start start 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.766+0000 7f1939176700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f19340fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.766+0000 7f1939176700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19340fc7b0 con 0x7f19340fbea0 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.767+0000 7f1932d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f19340fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.767+0000 7f1932d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f19340fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35324/0 (socket says 172.21.15.143:35324) 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.767+0000 7f1932d9d700 1 -- 172.21.15.143:0/860416436 learned_addr learned my addr 172.21.15.143:0/860416436 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.767+0000 7f1932d9d700 1 -- 172.21.15.143:0/860416436 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19340fc8f0 con 0x7f19340fbea0 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.767+0000 7f1932d9d700 1 --2- 172.21.15.143:0/860416436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f19340fc2c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f192800b5f0 tx=0x7f1928006ee0).ready entity=mon.0 client_cookie=634b3aa8b2c4dc4d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.767+0000 7f1931d9b700 1 -- 172.21.15.143:0/860416436 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f192800d600 con 0x7f19340fbea0 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.767+0000 7f1931d9b700 1 -- 172.21.15.143:0/860416436 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f192800dbf0 con 0x7f19340fbea0 2024-02-19T21:54:08.770 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.767+0000 7f1931d9b700 1 -- 172.21.15.143:0/860416436 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1928011cb0 con 0x7f19340fbea0 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.768+0000 7f1939176700 1 -- 172.21.15.143:0/860416436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 msgr2=0x7f19340fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.768+0000 7f1939176700 1 --2- 172.21.15.143:0/860416436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f19340fc2c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f192800b5f0 tx=0x7f1928006ee0).stop 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.768+0000 7f1939176700 1 -- 172.21.15.143:0/860416436 shutdown_connections 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.768+0000 7f1939176700 1 --2- 172.21.15.143:0/860416436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f19340fc2c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.768+0000 7f1939176700 1 -- 172.21.15.143:0/860416436 >> 172.21.15.143:0/860416436 conn(0x7f19340f76a0 msgr2=0x7f19340f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.768+0000 7f1939176700 1 -- 172.21.15.143:0/860416436 shutdown_connections 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.768+0000 7f1939176700 1 -- 172.21.15.143:0/860416436 wait complete. 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.768+0000 7f1939176700 1 Processor -- start 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.768+0000 7f1939176700 1 -- start start 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f1939176700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f193410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f1939176700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193410b080 con 0x7f19340fbea0 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f1932d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f193410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f1932d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f193410ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:35338/0 (socket says 172.21.15.143:35338) 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f1932d9d700 1 -- 172.21.15.143:0/3663114089 learned_addr learned my addr 172.21.15.143:0/3663114089 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f1932d9d700 1 -- 172.21.15.143:0/3663114089 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192800b200 con 0x7f19340fbea0 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f1932d9d700 1 --2- 172.21.15.143:0/3663114089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f193410ab90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f192800bf40 tx=0x7f1928006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:08.771 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f191b7fe700 1 -- 172.21.15.143:0/3663114089 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f192801b570 con 0x7f19340fbea0 2024-02-19T21:54:08.772 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f191b7fe700 1 -- 172.21.15.143:0/3663114089 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f192801bb60 con 0x7f19340fbea0 2024-02-19T21:54:08.772 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f1939176700 1 -- 172.21.15.143:0/3663114089 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193410b280 con 0x7f19340fbea0 2024-02-19T21:54:08.772 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f191b7fe700 1 -- 172.21.15.143:0/3663114089 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f192801aa20 con 0x7f19340fbea0 2024-02-19T21:54:08.772 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.769+0000 7f1939176700 1 -- 172.21.15.143:0/3663114089 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193410b710 con 0x7f19340fbea0 2024-02-19T21:54:08.772 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.770+0000 7f191b7fe700 1 -- 172.21.15.143:0/3663114089 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f192800f070 con 0x7f19340fbea0 2024-02-19T21:54:08.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.770+0000 7f191b7fe700 1 --2- 172.21.15.143:0/3663114089 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f192002ff20 0x7f19200323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:08.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.770+0000 7f191b7fe700 1 -- 172.21.15.143:0/3663114089 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1928041640 con 0x7f19340fbea0 2024-02-19T21:54:08.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.770+0000 7f1939176700 1 -- 172.21.15.143:0/3663114089 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1934045ad0 con 0x7f19340fbea0 2024-02-19T21:54:08.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.771+0000 7f193259c700 1 --2- 172.21.15.143:0/3663114089 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f192002ff20 0x7f19200323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:08.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.771+0000 7f193259c700 1 --2- 172.21.15.143:0/3663114089 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f192002ff20 0x7f19200323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f19240096a0 tx=0x7f1924006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:08.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.775+0000 7f191b7fe700 1 -- 172.21.15.143:0/3663114089 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1928016070 con 0x7f19340fbea0 2024-02-19T21:54:08.953 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.953+0000 7f1939176700 1 -- 172.21.15.143:0/3663114089 --> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f193410bf70 con 0x7f192002ff20 2024-02-19T21:54:08.957 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.957+0000 7f191b7fe700 1 -- 172.21.15.143:0/3663114089 <== mgr.14142 v2:172.21.15.143:6800/766735871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f193410bf70 con 0x7f192002ff20 2024-02-19T21:54:08.957 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-19T21:54:08.959 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.959+0000 7f1939176700 1 -- 172.21.15.143:0/3663114089 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f192002ff20 msgr2=0x7f19200323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:08.959 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.959+0000 7f1939176700 1 --2- 172.21.15.143:0/3663114089 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f192002ff20 0x7f19200323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f19240096a0 tx=0x7f1924006b40).stop 2024-02-19T21:54:08.959 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.959+0000 7f1939176700 1 -- 172.21.15.143:0/3663114089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 msgr2=0x7f193410ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:08.959 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.959+0000 7f1939176700 1 --2- 172.21.15.143:0/3663114089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f193410ab90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f192800bf40 tx=0x7f1928006ee0).stop 2024-02-19T21:54:08.960 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.959+0000 7f1939176700 1 -- 172.21.15.143:0/3663114089 shutdown_connections 2024-02-19T21:54:08.960 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.959+0000 7f1939176700 1 --2- 172.21.15.143:0/3663114089 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f192002ff20 0x7f19200323e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:08.960 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.959+0000 7f1939176700 1 --2- 172.21.15.143:0/3663114089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19340fbea0 0x7f193410ab90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:08.960 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.959+0000 7f1939176700 1 -- 172.21.15.143:0/3663114089 >> 172.21.15.143:0/3663114089 conn(0x7f19340f76a0 msgr2=0x7f19340f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:08.960 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.959+0000 7f1939176700 1 -- 172.21.15.143:0/3663114089 shutdown_connections 2024-02-19T21:54:08.960 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:08.959+0000 7f1939176700 1 -- 172.21.15.143:0/3663114089 wait complete. 2024-02-19T21:54:09.607 INFO:teuthology.orchestra.run.smithi143.stderr:Deploying alertmanager service with default placement... 2024-02-19T21:54:09.607 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-19T21:54:10.118 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.115+0000 7f4c84c8b700 1 Processor -- start 2024-02-19T21:54:10.118 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.115+0000 7f4c84c8b700 1 -- start start 2024-02-19T21:54:10.118 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.115+0000 7f4c84c8b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c800fe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:10.118 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.115+0000 7f4c84c8b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c800fedc0 con 0x7f4c800703f0 2024-02-19T21:54:10.118 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.115+0000 7f4c7f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c800fe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:10.118 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.115+0000 7f4c7f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c800fe8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52120/0 (socket says 172.21.15.143:52120) 2024-02-19T21:54:10.118 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.115+0000 7f4c7f7fe700 1 -- 172.21.15.143:0/4140207476 learned_addr learned my addr 172.21.15.143:0/4140207476 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.115+0000 7f4c7f7fe700 1 -- 172.21.15.143:0/4140207476 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c800fef00 con 0x7f4c800703f0 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.115+0000 7f4c7f7fe700 1 --2- 172.21.15.143:0/4140207476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c800fe8d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f4c7000b5f0 tx=0x7f4c70006ee0).ready entity=mon.0 client_cookie=96b1ba2301a21d3c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.116+0000 7f4c7e7fc700 1 -- 172.21.15.143:0/4140207476 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c7000d600 con 0x7f4c800703f0 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.116+0000 7f4c7e7fc700 1 -- 172.21.15.143:0/4140207476 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4c7000dbf0 con 0x7f4c800703f0 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.116+0000 7f4c7e7fc700 1 -- 172.21.15.143:0/4140207476 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c70011ce0 con 0x7f4c800703f0 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.116+0000 7f4c84c8b700 1 -- 172.21.15.143:0/4140207476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 msgr2=0x7f4c800fe8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.116+0000 7f4c84c8b700 1 --2- 172.21.15.143:0/4140207476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c800fe8d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f4c7000b5f0 tx=0x7f4c70006ee0).stop 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.116+0000 7f4c84c8b700 1 -- 172.21.15.143:0/4140207476 shutdown_connections 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.116+0000 7f4c84c8b700 1 --2- 172.21.15.143:0/4140207476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c800fe8d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.116+0000 7f4c84c8b700 1 -- 172.21.15.143:0/4140207476 >> 172.21.15.143:0/4140207476 conn(0x7f4c800f6390 msgr2=0x7f4c800f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.116+0000 7f4c84c8b700 1 -- 172.21.15.143:0/4140207476 shutdown_connections 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.116+0000 7f4c84c8b700 1 -- 172.21.15.143:0/4140207476 wait complete. 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.117+0000 7f4c84c8b700 1 Processor -- start 2024-02-19T21:54:10.119 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.117+0000 7f4c84c8b700 1 -- start start 2024-02-19T21:54:10.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.117+0000 7f4c84c8b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c8010a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:10.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.117+0000 7f4c84c8b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8010ae70 con 0x7f4c800703f0 2024-02-19T21:54:10.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.117+0000 7f4c7f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c8010a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:10.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.117+0000 7f4c7f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c8010a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52122/0 (socket says 172.21.15.143:52122) 2024-02-19T21:54:10.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.117+0000 7f4c7f7fe700 1 -- 172.21.15.143:0/2794331667 learned_addr learned my addr 172.21.15.143:0/2794331667 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:10.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.118+0000 7f4c7f7fe700 1 -- 172.21.15.143:0/2794331667 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7000b200 con 0x7f4c800703f0 2024-02-19T21:54:10.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.118+0000 7f4c7f7fe700 1 --2- 172.21.15.143:0/2794331667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c8010a980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f4c70011ed0 tx=0x7f4c70006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:10.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.118+0000 7f4c67fff700 1 -- 172.21.15.143:0/2794331667 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c7001b570 con 0x7f4c800703f0 2024-02-19T21:54:10.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.118+0000 7f4c67fff700 1 -- 172.21.15.143:0/2794331667 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4c7001bb60 con 0x7f4c800703f0 2024-02-19T21:54:10.120 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.118+0000 7f4c84c8b700 1 -- 172.21.15.143:0/2794331667 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8010b070 con 0x7f4c800703f0 2024-02-19T21:54:10.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.118+0000 7f4c67fff700 1 -- 172.21.15.143:0/2794331667 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c7001aad0 con 0x7f4c800703f0 2024-02-19T21:54:10.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.118+0000 7f4c84c8b700 1 -- 172.21.15.143:0/2794331667 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8010b500 con 0x7f4c800703f0 2024-02-19T21:54:10.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.119+0000 7f4c67fff700 1 -- 172.21.15.143:0/2794331667 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f4c7000f070 con 0x7f4c800703f0 2024-02-19T21:54:10.121 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.119+0000 7f4c84c8b700 1 -- 172.21.15.143:0/2794331667 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c80045ad0 con 0x7f4c800703f0 2024-02-19T21:54:10.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.119+0000 7f4c67fff700 1 --2- 172.21.15.143:0/2794331667 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f4c6c02ff70 0x7f4c6c032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:10.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.119+0000 7f4c67fff700 1 -- 172.21.15.143:0/2794331667 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4c700414c0 con 0x7f4c800703f0 2024-02-19T21:54:10.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.119+0000 7f4c7effd700 1 --2- 172.21.15.143:0/2794331667 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f4c6c02ff70 0x7f4c6c032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:10.122 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.119+0000 7f4c7effd700 1 --2- 172.21.15.143:0/2794331667 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f4c6c02ff70 0x7f4c6c032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4c740096a0 tx=0x7f4c74006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:10.123 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.123+0000 7f4c67fff700 1 -- 172.21.15.143:0/2794331667 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c70025980 con 0x7f4c800703f0 2024-02-19T21:54:10.298 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.299+0000 7f4c84c8b700 1 -- 172.21.15.143:0/2794331667 --> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c8010be20 con 0x7f4c6c02ff70 2024-02-19T21:54:10.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.305+0000 7f4c67fff700 1 -- 172.21.15.143:0/2794331667 <== mgr.14142 v2:172.21.15.143:6800/766735871 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f4c8010be20 con 0x7f4c6c02ff70 2024-02-19T21:54:10.305 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-19T21:54:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.307+0000 7f4c84c8b700 1 -- 172.21.15.143:0/2794331667 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f4c6c02ff70 msgr2=0x7f4c6c032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.307+0000 7f4c84c8b700 1 --2- 172.21.15.143:0/2794331667 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f4c6c02ff70 0x7f4c6c032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4c740096a0 tx=0x7f4c74006b40).stop 2024-02-19T21:54:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.307+0000 7f4c84c8b700 1 -- 172.21.15.143:0/2794331667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 msgr2=0x7f4c8010a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.307+0000 7f4c84c8b700 1 --2- 172.21.15.143:0/2794331667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c8010a980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f4c70011ed0 tx=0x7f4c70006ee0).stop 2024-02-19T21:54:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.307+0000 7f4c84c8b700 1 -- 172.21.15.143:0/2794331667 shutdown_connections 2024-02-19T21:54:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.307+0000 7f4c84c8b700 1 --2- 172.21.15.143:0/2794331667 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f4c6c02ff70 0x7f4c6c032430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.307+0000 7f4c84c8b700 1 --2- 172.21.15.143:0/2794331667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c800703f0 0x7f4c8010a980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.307+0000 7f4c84c8b700 1 -- 172.21.15.143:0/2794331667 >> 172.21.15.143:0/2794331667 conn(0x7f4c800f6390 msgr2=0x7f4c800f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:10.308 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.307+0000 7f4c84c8b700 1 -- 172.21.15.143:0/2794331667 shutdown_connections 2024-02-19T21:54:10.309 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:10.307+0000 7f4c84c8b700 1 -- 172.21.15.143:0/2794331667 wait complete. 2024-02-19T21:54:10.330 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:09 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:08.954829+0000 mgr.smithi143.bissbc (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:54:10.331 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:09 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:54:08.955490+0000 mgr.smithi143.bissbc (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-19T21:54:10.331 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:09 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:08.957860+0000 mon.smithi143 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:10.955 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-19T21:54:11.508 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.507+0000 7f79d6724700 1 Processor -- start 2024-02-19T21:54:11.508 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.507+0000 7f79d6724700 1 -- start start 2024-02-19T21:54:11.508 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.507+0000 7f79d6724700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d00fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:11.509 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.507+0000 7f79d6724700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d00fb6a0 con 0x7f79d00fad90 2024-02-19T21:54:11.509 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.508+0000 7f79cffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d00fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:11.509 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.508+0000 7f79cffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d00fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52130/0 (socket says 172.21.15.143:52130) 2024-02-19T21:54:11.509 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.508+0000 7f79cffff700 1 -- 172.21.15.143:0/150617759 learned_addr learned my addr 172.21.15.143:0/150617759 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:11.509 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.508+0000 7f79cffff700 1 -- 172.21.15.143:0/150617759 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d00fb7e0 con 0x7f79d00fad90 2024-02-19T21:54:11.509 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.508+0000 7f79cffff700 1 --2- 172.21.15.143:0/150617759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d00fb1b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f79b800b5f0 tx=0x7f79b8006ee0).ready entity=mon.0 client_cookie=93235e6f218e7b8e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:11.509 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.508+0000 7f79ceffd700 1 -- 172.21.15.143:0/150617759 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b800d600 con 0x7f79d00fad90 2024-02-19T21:54:11.509 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.508+0000 7f79ceffd700 1 -- 172.21.15.143:0/150617759 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f79b800dbf0 con 0x7f79d00fad90 2024-02-19T21:54:11.509 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.508+0000 7f79ceffd700 1 -- 172.21.15.143:0/150617759 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b8011cb0 con 0x7f79d00fad90 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.509+0000 7f79d6724700 1 -- 172.21.15.143:0/150617759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 msgr2=0x7f79d00fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.509+0000 7f79d6724700 1 --2- 172.21.15.143:0/150617759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d00fb1b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f79b800b5f0 tx=0x7f79b8006ee0).stop 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.509+0000 7f79d6724700 1 -- 172.21.15.143:0/150617759 shutdown_connections 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.509+0000 7f79d6724700 1 --2- 172.21.15.143:0/150617759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d00fb1b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.509+0000 7f79d6724700 1 -- 172.21.15.143:0/150617759 >> 172.21.15.143:0/150617759 conn(0x7f79d00f6530 msgr2=0x7f79d00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.509+0000 7f79d6724700 1 -- 172.21.15.143:0/150617759 shutdown_connections 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.509+0000 7f79d6724700 1 -- 172.21.15.143:0/150617759 wait complete. 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.510+0000 7f79d6724700 1 Processor -- start 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.510+0000 7f79d6724700 1 -- start start 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.510+0000 7f79d6724700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d0101040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.510+0000 7f79d6724700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d00ff4c0 con 0x7f79d00fad90 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.510+0000 7f79cffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d0101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:11.512 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.510+0000 7f79cffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d0101040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52144/0 (socket says 172.21.15.143:52144) 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.510+0000 7f79cffff700 1 -- 172.21.15.143:0/3257820672 learned_addr learned my addr 172.21.15.143:0/3257820672 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.510+0000 7f79cffff700 1 -- 172.21.15.143:0/3257820672 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b800b200 con 0x7f79d00fad90 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.511+0000 7f79cffff700 1 --2- 172.21.15.143:0/3257820672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d0101040 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f79b8011ea0 tx=0x7f79b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.511+0000 7f79ccff9700 1 -- 172.21.15.143:0/3257820672 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b801b570 con 0x7f79d00fad90 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.511+0000 7f79ccff9700 1 -- 172.21.15.143:0/3257820672 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f79b801bb60 con 0x7f79d00fad90 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.511+0000 7f79ccff9700 1 -- 172.21.15.143:0/3257820672 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b801a8d0 con 0x7f79d00fad90 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.511+0000 7f79d6724700 1 -- 172.21.15.143:0/3257820672 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d00ff660 con 0x7f79d00fad90 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.511+0000 7f79d6724700 1 -- 172.21.15.143:0/3257820672 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d00ffaf0 con 0x7f79d00fad90 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.512+0000 7f79ccff9700 1 -- 172.21.15.143:0/3257820672 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f79b800f070 con 0x7f79d00fad90 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.512+0000 7f79ccff9700 1 --2- 172.21.15.143:0/3257820672 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f79b002ff20 0x7f79b00323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.512+0000 7f79ccff9700 1 -- 172.21.15.143:0/3257820672 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f79b8041750 con 0x7f79d00fad90 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.512+0000 7f79d6724700 1 -- 172.21.15.143:0/3257820672 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d0045ad0 con 0x7f79d00fad90 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.512+0000 7f79cf7fe700 1 --2- 172.21.15.143:0/3257820672 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f79b002ff20 0x7f79b00323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:11.513 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.512+0000 7f79cf7fe700 1 --2- 172.21.15.143:0/3257820672 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f79b002ff20 0x7f79b00323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f79c00096a0 tx=0x7f79c0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:11.516 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.516+0000 7f79ccff9700 1 -- 172.21.15.143:0/3257820672 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f79b8016020 con 0x7f79d00fad90 2024-02-19T21:54:11.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:11 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:10.300410+0000 mgr.smithi143.bissbc (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:54:11.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:11 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:54:10.300919+0000 mgr.smithi143.bissbc (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-19T21:54:11.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:11 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:10.306017+0000 mon.smithi143 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.143:0/2629935619' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:11.667 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.667+0000 7f79d6724700 1 -- 172.21.15.143:0/3257820672 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f79d0104c00 con 0x7f79d00fad90 2024-02-19T21:54:11.669 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.670+0000 7f79ccff9700 1 -- 172.21.15.143:0/3257820672 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f79b8022d20 con 0x7f79d00fad90 2024-02-19T21:54:11.672 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.672+0000 7f79d6724700 1 -- 172.21.15.143:0/3257820672 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f79b002ff20 msgr2=0x7f79b00323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:11.673 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.672+0000 7f79d6724700 1 --2- 172.21.15.143:0/3257820672 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f79b002ff20 0x7f79b00323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f79c00096a0 tx=0x7f79c0006b40).stop 2024-02-19T21:54:11.673 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.672+0000 7f79d6724700 1 -- 172.21.15.143:0/3257820672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 msgr2=0x7f79d0101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:11.673 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.672+0000 7f79d6724700 1 --2- 172.21.15.143:0/3257820672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d0101040 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f79b8011ea0 tx=0x7f79b8006ee0).stop 2024-02-19T21:54:11.673 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.672+0000 7f79d6724700 1 -- 172.21.15.143:0/3257820672 shutdown_connections 2024-02-19T21:54:11.673 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.672+0000 7f79d6724700 1 --2- 172.21.15.143:0/3257820672 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f79b002ff20 0x7f79b00323e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:11.673 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.672+0000 7f79d6724700 1 --2- 172.21.15.143:0/3257820672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d00fad90 0x7f79d0101040 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:11.673 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.672+0000 7f79d6724700 1 -- 172.21.15.143:0/3257820672 >> 172.21.15.143:0/3257820672 conn(0x7f79d00f6530 msgr2=0x7f79d0070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:11.673 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.672+0000 7f79d6724700 1 -- 172.21.15.143:0/3257820672 shutdown_connections 2024-02-19T21:54:11.673 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:11.672+0000 7f79d6724700 1 -- 172.21.15.143:0/3257820672 wait complete. 2024-02-19T21:54:12.311 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-19T21:54:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.842+0000 7fb4204bd700 1 Processor -- start 2024-02-19T21:54:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.842+0000 7fb4204bd700 1 -- start start 2024-02-19T21:54:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.842+0000 7fb4204bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb4180fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.842+0000 7fb4204bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4180fef80 con 0x7fb4180703f0 2024-02-19T21:54:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.843+0000 7fb41e259700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb4180fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.843+0000 7fb41e259700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb4180fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52158/0 (socket says 172.21.15.143:52158) 2024-02-19T21:54:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.843+0000 7fb41e259700 1 -- 172.21.15.143:0/3466786260 learned_addr learned my addr 172.21.15.143:0/3466786260 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.843+0000 7fb41e259700 1 -- 172.21.15.143:0/3466786260 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4180ff0c0 con 0x7fb4180703f0 2024-02-19T21:54:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.843+0000 7fb41e259700 1 --2- 172.21.15.143:0/3466786260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb4180fea90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb40800b5f0 tx=0x7fb408006ee0).ready entity=mon.0 client_cookie=773cd4e9988aca8c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.843+0000 7fb41d257700 1 -- 172.21.15.143:0/3466786260 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb40800d600 con 0x7fb4180703f0 2024-02-19T21:54:12.891 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.843+0000 7fb41d257700 1 -- 172.21.15.143:0/3466786260 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb40800dbf0 con 0x7fb4180703f0 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.843+0000 7fb41d257700 1 -- 172.21.15.143:0/3466786260 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb408011ce0 con 0x7fb4180703f0 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.843+0000 7fb4204bd700 1 -- 172.21.15.143:0/3466786260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 msgr2=0x7fb4180fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.843+0000 7fb4204bd700 1 --2- 172.21.15.143:0/3466786260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb4180fea90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb40800b5f0 tx=0x7fb408006ee0).stop 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb4204bd700 1 -- 172.21.15.143:0/3466786260 shutdown_connections 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb4204bd700 1 --2- 172.21.15.143:0/3466786260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb4180fea90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb4204bd700 1 -- 172.21.15.143:0/3466786260 >> 172.21.15.143:0/3466786260 conn(0x7fb4180f6530 msgr2=0x7fb4180f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb4204bd700 1 -- 172.21.15.143:0/3466786260 shutdown_connections 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb4204bd700 1 -- 172.21.15.143:0/3466786260 wait complete. 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb4204bd700 1 Processor -- start 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb4204bd700 1 -- start start 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb4204bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb41810ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb4204bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41810b010 con 0x7fb4180703f0 2024-02-19T21:54:12.892 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb41e259700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb41810ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb41e259700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb41810ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52172/0 (socket says 172.21.15.143:52172) 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.844+0000 7fb41e259700 1 -- 172.21.15.143:0/1983784528 learned_addr learned my addr 172.21.15.143:0/1983784528 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.845+0000 7fb41e259700 1 -- 172.21.15.143:0/1983784528 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40800b200 con 0x7fb4180703f0 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.845+0000 7fb41e259700 1 --2- 172.21.15.143:0/1983784528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb41810ab20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb408011ed0 tx=0x7fb408006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.845+0000 7fb406ffd700 1 -- 172.21.15.143:0/1983784528 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb40801b570 con 0x7fb4180703f0 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.845+0000 7fb4204bd700 1 -- 172.21.15.143:0/1983784528 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41810b210 con 0x7fb4180703f0 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.845+0000 7fb406ffd700 1 -- 172.21.15.143:0/1983784528 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb40801bb60 con 0x7fb4180703f0 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.845+0000 7fb406ffd700 1 -- 172.21.15.143:0/1983784528 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb40801aad0 con 0x7fb4180703f0 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.845+0000 7fb4204bd700 1 -- 172.21.15.143:0/1983784528 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41810b6a0 con 0x7fb4180703f0 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.846+0000 7fb406ffd700 1 -- 172.21.15.143:0/1983784528 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb40800f070 con 0x7fb4180703f0 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.846+0000 7fb406ffd700 1 --2- 172.21.15.143:0/1983784528 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fb40002ff20 0x7fb4000323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.846+0000 7fb406ffd700 1 -- 172.21.15.143:0/1983784528 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb408041510 con 0x7fb4180703f0 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.846+0000 7fb4204bd700 1 -- 172.21.15.143:0/1983784528 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb418045ad0 con 0x7fb4180703f0 2024-02-19T21:54:12.893 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.846+0000 7fb41da58700 1 --2- 172.21.15.143:0/1983784528 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fb40002ff20 0x7fb4000323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:12.894 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.846+0000 7fb41da58700 1 --2- 172.21.15.143:0/1983784528 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fb40002ff20 0x7fb4000323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb4140096a0 tx=0x7fb414006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:12.894 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:12.850+0000 7fb406ffd700 1 -- 172.21.15.143:0/1983784528 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb408016070 con 0x7fb4180703f0 2024-02-19T21:54:13.001 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.001+0000 7fb4204bd700 1 -- 172.21.15.143:0/1983784528 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb4180ffe00 con 0x7fb4180703f0 2024-02-19T21:54:13.003 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.004+0000 7fb406ffd700 1 -- 172.21.15.143:0/1983784528 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb408022310 con 0x7fb4180703f0 2024-02-19T21:54:13.006 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.006+0000 7fb4204bd700 1 -- 172.21.15.143:0/1983784528 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fb40002ff20 msgr2=0x7fb4000323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:13.006 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.006+0000 7fb4204bd700 1 --2- 172.21.15.143:0/1983784528 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fb40002ff20 0x7fb4000323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb4140096a0 tx=0x7fb414006b40).stop 2024-02-19T21:54:13.007 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.006+0000 7fb4204bd700 1 -- 172.21.15.143:0/1983784528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 msgr2=0x7fb41810ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:13.007 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.006+0000 7fb4204bd700 1 --2- 172.21.15.143:0/1983784528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb41810ab20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb408011ed0 tx=0x7fb408006ee0).stop 2024-02-19T21:54:13.007 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.006+0000 7fb4204bd700 1 -- 172.21.15.143:0/1983784528 shutdown_connections 2024-02-19T21:54:13.007 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.006+0000 7fb4204bd700 1 --2- 172.21.15.143:0/1983784528 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fb40002ff20 0x7fb4000323e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:13.007 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.006+0000 7fb4204bd700 1 --2- 172.21.15.143:0/1983784528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180703f0 0x7fb41810ab20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:13.007 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.006+0000 7fb4204bd700 1 -- 172.21.15.143:0/1983784528 >> 172.21.15.143:0/1983784528 conn(0x7fb4180f6530 msgr2=0x7fb4180f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:13.007 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.006+0000 7fb4204bd700 1 -- 172.21.15.143:0/1983784528 shutdown_connections 2024-02-19T21:54:13.007 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:13.006+0000 7fb4204bd700 1 -- 172.21.15.143:0/1983784528 wait complete. 2024-02-19T21:54:13.027 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:11.671006+0000 mon.smithi143 (mon.0) 82 : audit [INF] from='client.? 172.21.15.143:0/3257820672' entity='client.admin' 2024-02-19T21:54:14.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:13.005052+0000 mon.smithi143 (mon.0) 83 : audit [INF] from='client.? 172.21.15.143:0/1983784528' entity='client.admin' 2024-02-19T21:54:14.380 INFO:teuthology.orchestra.run.smithi143.stderr:Enabling the dashboard module... 2024-02-19T21:54:14.380 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-19T21:54:15.154 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.152+0000 7fed4ca48700 1 Processor -- start 2024-02-19T21:54:15.154 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.152+0000 7fed4ca48700 1 -- start start 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.152+0000 7fed4ca48700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed480fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.153+0000 7fed4ca48700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed480fef80 con 0x7fed480703f0 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.153+0000 7fed4659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed480fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.153+0000 7fed4659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed480fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52184/0 (socket says 172.21.15.143:52184) 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.153+0000 7fed4659c700 1 -- 172.21.15.143:0/95815375 learned_addr learned my addr 172.21.15.143:0/95815375 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.153+0000 7fed4659c700 1 -- 172.21.15.143:0/95815375 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed480ff0c0 con 0x7fed480703f0 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.153+0000 7fed4659c700 1 --2- 172.21.15.143:0/95815375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed480fea90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fed3c00b5f0 tx=0x7fed3c006ee0).ready entity=mon.0 client_cookie=a390429399c76e1a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.153+0000 7fed4559a700 1 -- 172.21.15.143:0/95815375 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed3c00d600 con 0x7fed480703f0 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.153+0000 7fed4559a700 1 -- 172.21.15.143:0/95815375 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fed3c00dbf0 con 0x7fed480703f0 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.153+0000 7fed4559a700 1 -- 172.21.15.143:0/95815375 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed3c011ce0 con 0x7fed480703f0 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.154+0000 7fed4ca48700 1 -- 172.21.15.143:0/95815375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 msgr2=0x7fed480fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.154+0000 7fed4ca48700 1 --2- 172.21.15.143:0/95815375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed480fea90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fed3c00b5f0 tx=0x7fed3c006ee0).stop 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.154+0000 7fed4ca48700 1 -- 172.21.15.143:0/95815375 shutdown_connections 2024-02-19T21:54:15.155 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.154+0000 7fed4ca48700 1 --2- 172.21.15.143:0/95815375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed480fea90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:15.156 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.154+0000 7fed4ca48700 1 -- 172.21.15.143:0/95815375 >> 172.21.15.143:0/95815375 conn(0x7fed480f6530 msgr2=0x7fed480f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:15.156 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.154+0000 7fed4ca48700 1 -- 172.21.15.143:0/95815375 shutdown_connections 2024-02-19T21:54:15.156 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.154+0000 7fed4ca48700 1 -- 172.21.15.143:0/95815375 wait complete. 2024-02-19T21:54:15.156 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.155+0000 7fed4ca48700 1 Processor -- start 2024-02-19T21:54:15.156 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.155+0000 7fed4ca48700 1 -- start start 2024-02-19T21:54:15.156 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.155+0000 7fed4ca48700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed4810ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:15.156 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.155+0000 7fed4ca48700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4810b010 con 0x7fed480703f0 2024-02-19T21:54:15.158 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.155+0000 7fed4659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed4810ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:15.158 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.155+0000 7fed4659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed4810ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52190/0 (socket says 172.21.15.143:52190) 2024-02-19T21:54:15.158 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.155+0000 7fed4659c700 1 -- 172.21.15.143:0/3837343444 learned_addr learned my addr 172.21.15.143:0/3837343444 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:15.158 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.155+0000 7fed4659c700 1 -- 172.21.15.143:0/3837343444 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c00b200 con 0x7fed480703f0 2024-02-19T21:54:15.158 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.155+0000 7fed4659c700 1 --2- 172.21.15.143:0/3837343444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed4810ab20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fed3c011ed0 tx=0x7fed3c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:15.158 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.155+0000 7fed36ffd700 1 -- 172.21.15.143:0/3837343444 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed3c01b570 con 0x7fed480703f0 2024-02-19T21:54:15.158 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.156+0000 7fed4ca48700 1 -- 172.21.15.143:0/3837343444 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4810b210 con 0x7fed480703f0 2024-02-19T21:54:15.158 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.156+0000 7fed36ffd700 1 -- 172.21.15.143:0/3837343444 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fed3c01bb60 con 0x7fed480703f0 2024-02-19T21:54:15.158 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.156+0000 7fed36ffd700 1 -- 172.21.15.143:0/3837343444 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed3c01aad0 con 0x7fed480703f0 2024-02-19T21:54:15.158 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.156+0000 7fed4ca48700 1 -- 172.21.15.143:0/3837343444 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4810b6a0 con 0x7fed480703f0 2024-02-19T21:54:15.158 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.156+0000 7fed36ffd700 1 -- 172.21.15.143:0/3837343444 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fed3c00f070 con 0x7fed480703f0 2024-02-19T21:54:15.159 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.157+0000 7fed36ffd700 1 --2- 172.21.15.143:0/3837343444 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fed2c02ff20 0x7fed2c0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:15.159 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.157+0000 7fed36ffd700 1 -- 172.21.15.143:0/3837343444 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fed3c041510 con 0x7fed480703f0 2024-02-19T21:54:15.159 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.157+0000 7fed45d9b700 1 --2- 172.21.15.143:0/3837343444 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fed2c02ff20 0x7fed2c0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:15.159 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.157+0000 7fed4ca48700 1 -- 172.21.15.143:0/3837343444 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed48045ad0 con 0x7fed480703f0 2024-02-19T21:54:15.159 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.157+0000 7fed45d9b700 1 --2- 172.21.15.143:0/3837343444 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fed2c02ff20 0x7fed2c0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fed380096a0 tx=0x7fed38006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:15.161 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.162+0000 7fed36ffd700 1 -- 172.21.15.143:0/3837343444 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed3c016070 con 0x7fed480703f0 2024-02-19T21:54:15.351 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.350+0000 7fed4ca48700 1 -- 172.21.15.143:0/3837343444 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fed48104b30 con 0x7fed480703f0 2024-02-19T21:54:15.402 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.402+0000 7fed36ffd700 1 -- 172.21.15.143:0/3837343444 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fed3c01ac30 con 0x7fed480703f0 2024-02-19T21:54:15.403 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.404+0000 7fed36ffd700 1 -- 172.21.15.143:0/3837343444 <== mon.0 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7fed3c02e5c0 con 0x7fed480703f0 2024-02-19T21:54:15.406 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.406+0000 7fed4ca48700 1 -- 172.21.15.143:0/3837343444 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fed2c02ff20 msgr2=0x7fed2c0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:15.406 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.406+0000 7fed4ca48700 1 --2- 172.21.15.143:0/3837343444 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fed2c02ff20 0x7fed2c0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fed380096a0 tx=0x7fed38006b40).stop 2024-02-19T21:54:15.406 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.406+0000 7fed4ca48700 1 -- 172.21.15.143:0/3837343444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 msgr2=0x7fed4810ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:15.407 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.406+0000 7fed4ca48700 1 --2- 172.21.15.143:0/3837343444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed4810ab20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fed3c011ed0 tx=0x7fed3c006ee0).stop 2024-02-19T21:54:15.407 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.406+0000 7fed4ca48700 1 -- 172.21.15.143:0/3837343444 shutdown_connections 2024-02-19T21:54:15.407 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.406+0000 7fed4ca48700 1 --2- 172.21.15.143:0/3837343444 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7fed2c02ff20 0x7fed2c0323e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:15.407 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.406+0000 7fed4ca48700 1 --2- 172.21.15.143:0/3837343444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed480703f0 0x7fed4810ab20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:15.407 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.406+0000 7fed4ca48700 1 -- 172.21.15.143:0/3837343444 >> 172.21.15.143:0/3837343444 conn(0x7fed480f6530 msgr2=0x7fed480f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:15.407 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.406+0000 7fed4ca48700 1 -- 172.21.15.143:0/3837343444 shutdown_connections 2024-02-19T21:54:15.407 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:15.406+0000 7fed4ca48700 1 -- 172.21.15.143:0/3837343444 wait complete. 2024-02-19T21:54:15.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:15 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:15.351310+0000 mon.smithi143 (mon.0) 84 : audit [INF] from='client.? 172.21.15.143:0/3837343444' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-19T21:54:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T21:54:16.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:15.404810+0000 mon.smithi143 (mon.0) 85 : audit [INF] from='client.? 172.21.15.143:0/3837343444' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-19T21:54:16.662 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:15.404869+0000 mon.smithi143 (mon.0) 86 : cluster [DBG] mgrmap e13: smithi143.bissbc(active, since 10s) 2024-02-19T21:54:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.766+0000 7f3c96787700 1 Processor -- start 2024-02-19T21:54:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.766+0000 7f3c96787700 1 -- start start 2024-02-19T21:54:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.767+0000 7f3c96787700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c90070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.767+0000 7f3c96787700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c90072940 con 0x7f3c90072520 2024-02-19T21:54:16.772 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.767+0000 7f3c8ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c90070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.767+0000 7f3c8ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c90070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52216/0 (socket says 172.21.15.143:52216) 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.767+0000 7f3c8ffff700 1 -- 172.21.15.143:0/4264023667 learned_addr learned my addr 172.21.15.143:0/4264023667 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.767+0000 7f3c8ffff700 1 -- 172.21.15.143:0/4264023667 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c90070fb0 con 0x7f3c90072520 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.767+0000 7f3c8ffff700 1 --2- 172.21.15.143:0/4264023667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c90070ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3c8000b5f0 tx=0x7f3c80006ee0).ready entity=mon.0 client_cookie=cf5d5746b4a7b312 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.767+0000 7f3c8effd700 1 -- 172.21.15.143:0/4264023667 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c8000d600 con 0x7f3c90072520 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.767+0000 7f3c8effd700 1 -- 172.21.15.143:0/4264023667 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3c8000dbf0 con 0x7f3c90072520 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.767+0000 7f3c8effd700 1 -- 172.21.15.143:0/4264023667 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c80011cb0 con 0x7f3c90072520 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.768+0000 7f3c96787700 1 -- 172.21.15.143:0/4264023667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 msgr2=0x7f3c90070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.768+0000 7f3c96787700 1 --2- 172.21.15.143:0/4264023667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c90070ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3c8000b5f0 tx=0x7f3c80006ee0).stop 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.768+0000 7f3c96787700 1 -- 172.21.15.143:0/4264023667 shutdown_connections 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.768+0000 7f3c96787700 1 --2- 172.21.15.143:0/4264023667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c90070ac0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.768+0000 7f3c96787700 1 -- 172.21.15.143:0/4264023667 >> 172.21.15.143:0/4264023667 conn(0x7f3c900f76a0 msgr2=0x7f3c900f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.768+0000 7f3c96787700 1 -- 172.21.15.143:0/4264023667 shutdown_connections 2024-02-19T21:54:16.773 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.768+0000 7f3c96787700 1 -- 172.21.15.143:0/4264023667 wait complete. 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.769+0000 7f3c96787700 1 Processor -- start 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.769+0000 7f3c96787700 1 -- start start 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.769+0000 7f3c96787700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c9010ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.769+0000 7f3c96787700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9010d330 con 0x7f3c90072520 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.770+0000 7f3c8ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c9010ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.770+0000 7f3c8ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c9010ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52226/0 (socket says 172.21.15.143:52226) 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.770+0000 7f3c8ffff700 1 -- 172.21.15.143:0/140787948 learned_addr learned my addr 172.21.15.143:0/140787948 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.770+0000 7f3c8ffff700 1 -- 172.21.15.143:0/140787948 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8000b200 con 0x7f3c90072520 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.770+0000 7f3c8ffff700 1 --2- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c9010ce40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f3c8000dfc0 tx=0x7f3c80006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.770+0000 7f3c8cff9700 1 -- 172.21.15.143:0/140787948 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c8001b410 con 0x7f3c90072520 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.770+0000 7f3c8cff9700 1 -- 172.21.15.143:0/140787948 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3c8001ba00 con 0x7f3c90072520 2024-02-19T21:54:16.774 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.770+0000 7f3c96787700 1 -- 172.21.15.143:0/140787948 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9010d530 con 0x7f3c90072520 2024-02-19T21:54:16.775 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.770+0000 7f3c8cff9700 1 -- 172.21.15.143:0/140787948 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c8001a990 con 0x7f3c90072520 2024-02-19T21:54:16.775 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.770+0000 7f3c96787700 1 -- 172.21.15.143:0/140787948 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9010e140 con 0x7f3c90072520 2024-02-19T21:54:16.775 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.771+0000 7f3c8cff9700 1 -- 172.21.15.143:0/140787948 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3c8000f070 con 0x7f3c90072520 2024-02-19T21:54:16.775 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.771+0000 7f3c8cff9700 1 --2- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f3c7c02ff70 0x7f3c7c032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:16.775 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.771+0000 7f3c8cff9700 1 -- 172.21.15.143:0/140787948 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f3c80041490 con 0x7f3c90072520 2024-02-19T21:54:16.775 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.771+0000 7f3c8f7fe700 1 -- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f3c7c02ff70 msgr2=0x7f3c7c032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/766735871 2024-02-19T21:54:16.775 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.771+0000 7f3c8f7fe700 1 --2- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f3c7c02ff70 0x7f3c7c032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T21:54:16.775 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.771+0000 7f3c96787700 1 -- 172.21.15.143:0/140787948 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c90059100 con 0x7f3c90072520 2024-02-19T21:54:16.776 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.776+0000 7f3c8cff9700 1 -- 172.21.15.143:0/140787948 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c80016070 con 0x7f3c90072520 2024-02-19T21:54:16.972 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.970+0000 7f3c96787700 1 -- 172.21.15.143:0/140787948 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3c90102080 con 0x7f3c90072520 2024-02-19T21:54:16.972 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.971+0000 7f3c8cff9700 1 -- 172.21.15.143:0/140787948 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f3c80022b90 con 0x7f3c90072520 2024-02-19T21:54:16.972 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: { 2024-02-19T21:54:16.972 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-19T21:54:16.972 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "available": true, 2024-02-19T21:54:16.972 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "active_name": "smithi143.bissbc", 2024-02-19T21:54:16.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T21:54:16.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: } 2024-02-19T21:54:16.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.972+0000 7f3c8f7fe700 1 -- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f3c7c02ff70 msgr2=0x7f3c7c032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/766735871 2024-02-19T21:54:16.973 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.972+0000 7f3c8f7fe700 1 --2- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f3c7c02ff70 0x7f3c7c032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T21:54:16.974 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.973+0000 7f3c96787700 1 -- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f3c7c02ff70 msgr2=0x7f3c7c032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:54:16.974 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.973+0000 7f3c96787700 1 --2- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f3c7c02ff70 0x7f3c7c032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:54:16.974 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.973+0000 7f3c96787700 1 -- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 msgr2=0x7f3c9010ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:16.974 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.973+0000 7f3c96787700 1 --2- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c9010ce40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f3c8000dfc0 tx=0x7f3c80006ee0).stop 2024-02-19T21:54:16.975 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.973+0000 7f3c96787700 1 -- 172.21.15.143:0/140787948 shutdown_connections 2024-02-19T21:54:16.975 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.973+0000 7f3c96787700 1 --2- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f3c7c02ff70 0x7f3c7c032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:54:16.975 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.973+0000 7f3c96787700 1 --2- 172.21.15.143:0/140787948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c90072520 0x7f3c9010ce40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:16.975 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.973+0000 7f3c96787700 1 -- 172.21.15.143:0/140787948 >> 172.21.15.143:0/140787948 conn(0x7f3c900f76a0 msgr2=0x7f3c900f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:16.975 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.973+0000 7f3c96787700 1 -- 172.21.15.143:0/140787948 shutdown_connections 2024-02-19T21:54:16.975 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:16.973+0000 7f3c96787700 1 -- 172.21.15.143:0/140787948 wait complete. 2024-02-19T21:54:17.607 INFO:teuthology.orchestra.run.smithi143.stderr:Waiting for the mgr to restart... 2024-02-19T21:54:17.607 INFO:teuthology.orchestra.run.smithi143.stderr:Waiting for mgr epoch 13... 2024-02-19T21:54:17.608 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T21:54:17.610 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:17 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:16.971978+0000 mon.smithi143 (mon.0) 87 : audit [DBG] from='client.? 172.21.15.143:0/140787948' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T21:54:18.197 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.194+0000 7f78c1cc3700 1 Processor -- start 2024-02-19T21:54:18.197 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.194+0000 7f78c1cc3700 1 -- start start 2024-02-19T21:54:18.197 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.195+0000 7f78c1cc3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:18.197 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.195+0000 7f78c1cc3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc0fef80 con 0x7f78bc0703f0 2024-02-19T21:54:18.197 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.195+0000 7f78bb7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:18.197 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.195+0000 7f78bb7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52234/0 (socket says 172.21.15.143:52234) 2024-02-19T21:54:18.197 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.195+0000 7f78bb7fe700 1 -- 172.21.15.143:0/2190868456 learned_addr learned my addr 172.21.15.143:0/2190868456 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:18.197 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.195+0000 7f78bb7fe700 1 -- 172.21.15.143:0/2190868456 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc0ff0c0 con 0x7f78bc0703f0 2024-02-19T21:54:18.197 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.195+0000 7f78bb7fe700 1 --2- 172.21.15.143:0/2190868456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fea90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f78b000b5f0 tx=0x7f78b0006ee0).ready entity=mon.0 client_cookie=7642e1c8e8e0e0a7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:18.197 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.195+0000 7f78ba7fc700 1 -- 172.21.15.143:0/2190868456 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78b000d600 con 0x7f78bc0703f0 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.196+0000 7f78ba7fc700 1 -- 172.21.15.143:0/2190868456 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f78b000dbf0 con 0x7f78bc0703f0 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.196+0000 7f78ba7fc700 1 -- 172.21.15.143:0/2190868456 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78b0011ce0 con 0x7f78bc0703f0 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.196+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2190868456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 msgr2=0x7f78bc0fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.196+0000 7f78c1cc3700 1 --2- 172.21.15.143:0/2190868456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fea90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f78b000b5f0 tx=0x7f78b0006ee0).stop 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.196+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2190868456 shutdown_connections 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.196+0000 7f78c1cc3700 1 --2- 172.21.15.143:0/2190868456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fea90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.196+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2190868456 >> 172.21.15.143:0/2190868456 conn(0x7f78bc0f6530 msgr2=0x7f78bc0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.196+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2190868456 shutdown_connections 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.196+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2190868456 wait complete. 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.197+0000 7f78c1cc3700 1 Processor -- start 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.197+0000 7f78c1cc3700 1 -- start start 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.197+0000 7f78c1cc3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:18.198 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.197+0000 7f78c1cc3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc0fa990 con 0x7f78bc0703f0 2024-02-19T21:54:18.200 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.197+0000 7f78bb7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:18.200 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.197+0000 7f78bb7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:52250/0 (socket says 172.21.15.143:52250) 2024-02-19T21:54:18.200 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.197+0000 7f78bb7fe700 1 -- 172.21.15.143:0/2729787668 learned_addr learned my addr 172.21.15.143:0/2729787668 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:18.200 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.197+0000 7f78bb7fe700 1 -- 172.21.15.143:0/2729787668 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b000b200 con 0x7f78bc0703f0 2024-02-19T21:54:18.200 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.198+0000 7f78bb7fe700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fc510 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f78b0011ed0 tx=0x7f78b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:18.200 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.198+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78b001b570 con 0x7f78bc0703f0 2024-02-19T21:54:18.200 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.198+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2729787668 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78bc0fab30 con 0x7f78bc0703f0 2024-02-19T21:54:18.200 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.198+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f78b001bb60 con 0x7f78bc0703f0 2024-02-19T21:54:18.200 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.198+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78b001aad0 con 0x7f78bc0703f0 2024-02-19T21:54:18.201 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.198+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2729787668 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78bc0fafc0 con 0x7f78bc0703f0 2024-02-19T21:54:18.201 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.199+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f78b000f070 con 0x7f78bc0703f0 2024-02-19T21:54:18.201 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.199+0000 7f78a3fff700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 0x7f78a8032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:18.201 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.199+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 --> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78a8032af0 con 0x7f78a802ff70 2024-02-19T21:54:18.201 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.199+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f78b0041510 con 0x7f78bc0703f0 2024-02-19T21:54:18.201 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.199+0000 7f78baffd700 1 -- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 msgr2=0x7f78a8032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/766735871 2024-02-19T21:54:18.201 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.200+0000 7f78baffd700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 0x7f78a8032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T21:54:18.401 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.401+0000 7f78baffd700 1 -- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 msgr2=0x7f78a8032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/766735871 2024-02-19T21:54:18.401 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.401+0000 7f78baffd700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 0x7f78a8032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T21:54:18.802 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.802+0000 7f78baffd700 1 -- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 msgr2=0x7f78a8032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/766735871 2024-02-19T21:54:18.802 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:18.802+0000 7f78baffd700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 0x7f78a8032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T21:54:19.603 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:19.603+0000 7f78baffd700 1 -- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 msgr2=0x7f78a8032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:6800/766735871 2024-02-19T21:54:19.603 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:19.603+0000 7f78baffd700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 0x7f78a8032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T21:54:20.438 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:20.437+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f78b001ac30 con 0x7f78bc0703f0 2024-02-19T21:54:20.438 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:20.438+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 msgr2=0x7f78a8032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:54:20.438 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:20.438+0000 7f78a3fff700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 0x7f78a8032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:54:21.443 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.441+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f78b0041050 con 0x7f78bc0703f0 2024-02-19T21:54:21.443 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.441+0000 7f78a3fff700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f78a8033c00 0x7f78a8035ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:21.443 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.441+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 --> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78a8032af0 con 0x7f78a8033c00 2024-02-19T21:54:21.443 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.441+0000 7f78baffd700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f78a8033c00 0x7f78a8035ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:21.443 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.442+0000 7f78baffd700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f78a8033c00 0x7f78a8035ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f78ac0036d0 tx=0x7f78ac006ca0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:21.443 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.442+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 <== mgr.14162 v2:172.21.15.143:6800/2497069397 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f78a8032af0 con 0x7f78a8033c00 2024-02-19T21:54:21.447 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.448+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2729787668 --> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f78bc059100 con 0x7f78a8033c00 2024-02-19T21:54:21.449 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.448+0000 7f78a3fff700 1 -- 172.21.15.143:0/2729787668 <== mgr.14162 v2:172.21.15.143:6800/2497069397 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f78bc059100 con 0x7f78a8033c00 2024-02-19T21:54:21.449 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: { 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: } 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.449+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f78a8033c00 msgr2=0x7f78a8035ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.449+0000 7f78c1cc3700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f78a8033c00 0x7f78a8035ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f78ac0036d0 tx=0x7f78ac006ca0).stop 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.449+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 msgr2=0x7f78bc0fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.449+0000 7f78c1cc3700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fc510 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f78b0011ed0 tx=0x7f78b0006ee0).stop 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.449+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2729787668 shutdown_connections 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.449+0000 7f78c1cc3700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f78a8033c00 0x7f78a8035ff0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.449+0000 7f78c1cc3700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78bc0703f0 0x7f78bc0fc510 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.449+0000 7f78c1cc3700 1 --2- 172.21.15.143:0/2729787668 >> [v2:172.21.15.143:6800/766735871,v1:172.21.15.143:6801/766735871] conn(0x7f78a802ff70 0x7f78a8032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.449+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2729787668 >> 172.21.15.143:0/2729787668 conn(0x7f78bc0f6530 msgr2=0x7f78bc0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.449+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2729787668 shutdown_connections 2024-02-19T21:54:21.450 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:21.449+0000 7f78c1cc3700 1 -- 172.21.15.143:0/2729787668 wait complete. 2024-02-19T21:54:21.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:20.386526+0000 mon.smithi143 (mon.0) 88 : cluster [INF] Active manager daemon smithi143.bissbc restarted 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:20.386946+0000 mon.smithi143 (mon.0) 89 : cluster [INF] Activating manager daemon smithi143.bissbc 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:20.388567+0000 mon.smithi143 (mon.0) 90 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:20.439628+0000 mon.smithi143 (mon.0) 91 : cluster [DBG] mgrmap e14: smithi143.bissbc(active, starting, since 0.0527676s) 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:20.440316+0000 mon.smithi143 (mon.0) 92 : audit [DBG] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:20.440836+0000 mon.smithi143 (mon.0) 93 : audit [DBG] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mgr metadata", "who": "smithi143.bissbc", "id": "smithi143.bissbc"}]: dispatch 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:20.441618+0000 mon.smithi143 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:20.441808+0000 mon.smithi143 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:20.441985+0000 mon.smithi143 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:20.443931+0000 mon.smithi143 (mon.0) 97 : cluster [INF] Manager daemon smithi143.bissbc is now available 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:20.541129+0000 mon.smithi143 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:20.544201+0000 mon.smithi143 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:20.557839+0000 mon.smithi143 (mon.0) 100 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:54:21.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:20.559875+0000 mon.smithi143 (mon.0) 101 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T21:54:22.065 INFO:teuthology.orchestra.run.smithi143.stderr:mgr epoch 13 is available 2024-02-19T21:54:22.065 INFO:teuthology.orchestra.run.smithi143.stderr:Generating a dashboard self-signed certificate... 2024-02-19T21:54:22.065 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-19T21:54:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.617+0000 7f9a89381700 1 Processor -- start 2024-02-19T21:54:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.617+0000 7f9a89381700 1 -- start start 2024-02-19T21:54:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.617+0000 7f9a89381700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a840fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.617+0000 7f9a89381700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a840fc7b0 con 0x7f9a840fbea0 2024-02-19T21:54:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.617+0000 7f9a82ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a840fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.617+0000 7f9a82ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a840fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33520/0 (socket says 172.21.15.143:33520) 2024-02-19T21:54:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.617+0000 7f9a82ffd700 1 -- 172.21.15.143:0/2786658482 learned_addr learned my addr 172.21.15.143:0/2786658482 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.617+0000 7f9a82ffd700 1 -- 172.21.15.143:0/2786658482 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a840fc8f0 con 0x7f9a840fbea0 2024-02-19T21:54:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.617+0000 7f9a82ffd700 1 --2- 172.21.15.143:0/2786658482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a840fc2c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f9a7400b5f0 tx=0x7f9a74006ee0).ready entity=mon.0 client_cookie=23639d0b3428adc7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.617+0000 7f9a81ffb700 1 -- 172.21.15.143:0/2786658482 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a7400d600 con 0x7f9a840fbea0 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.618+0000 7f9a81ffb700 1 -- 172.21.15.143:0/2786658482 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9a7400dbf0 con 0x7f9a840fbea0 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.618+0000 7f9a81ffb700 1 -- 172.21.15.143:0/2786658482 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a74011cb0 con 0x7f9a840fbea0 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.618+0000 7f9a89381700 1 -- 172.21.15.143:0/2786658482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 msgr2=0x7f9a840fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.618+0000 7f9a89381700 1 --2- 172.21.15.143:0/2786658482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a840fc2c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f9a7400b5f0 tx=0x7f9a74006ee0).stop 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.618+0000 7f9a89381700 1 -- 172.21.15.143:0/2786658482 shutdown_connections 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.618+0000 7f9a89381700 1 --2- 172.21.15.143:0/2786658482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a840fc2c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.618+0000 7f9a89381700 1 -- 172.21.15.143:0/2786658482 >> 172.21.15.143:0/2786658482 conn(0x7f9a840f76a0 msgr2=0x7f9a840f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.618+0000 7f9a89381700 1 -- 172.21.15.143:0/2786658482 shutdown_connections 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.618+0000 7f9a89381700 1 -- 172.21.15.143:0/2786658482 wait complete. 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a89381700 1 Processor -- start 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a89381700 1 -- start start 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a89381700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a8410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a89381700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8410b080 con 0x7f9a840fbea0 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a82ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a8410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:22.646 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a82ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a8410ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33530/0 (socket says 172.21.15.143:33530) 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a82ffd700 1 -- 172.21.15.143:0/202485433 learned_addr learned my addr 172.21.15.143:0/202485433 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a82ffd700 1 -- 172.21.15.143:0/202485433 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7400b200 con 0x7f9a840fbea0 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a82ffd700 1 --2- 172.21.15.143:0/202485433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a8410ab90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f9a7400bf40 tx=0x7f9a74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a6b7fe700 1 -- 172.21.15.143:0/202485433 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a7401b570 con 0x7f9a840fbea0 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a6b7fe700 1 -- 172.21.15.143:0/202485433 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9a7401bb60 con 0x7f9a840fbea0 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a6b7fe700 1 -- 172.21.15.143:0/202485433 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a7401aa20 con 0x7f9a840fbea0 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a89381700 1 -- 172.21.15.143:0/202485433 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8410b280 con 0x7f9a840fbea0 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.619+0000 7f9a89381700 1 -- 172.21.15.143:0/202485433 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8410b710 con 0x7f9a840fbea0 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.620+0000 7f9a6b7fe700 1 -- 172.21.15.143:0/202485433 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f9a7400f070 con 0x7f9a840fbea0 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.620+0000 7f9a6b7fe700 1 --2- 172.21.15.143:0/202485433 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9a7002ffc0 0x7f9a70032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.620+0000 7f9a6b7fe700 1 -- 172.21.15.143:0/202485433 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a740416b0 con 0x7f9a840fbea0 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.620+0000 7f9a89381700 1 -- 172.21.15.143:0/202485433 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a84045ad0 con 0x7f9a840fbea0 2024-02-19T21:54:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.620+0000 7f9a827fc700 1 --2- 172.21.15.143:0/202485433 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9a7002ffc0 0x7f9a70032480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:22.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.621+0000 7f9a827fc700 1 --2- 172.21.15.143:0/202485433 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9a7002ffc0 0x7f9a70032480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9a780096a0 tx=0x7f9a78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:22.648 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.625+0000 7f9a6b7fe700 1 -- 172.21.15.143:0/202485433 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a74016070 con 0x7f9a840fbea0 2024-02-19T21:54:22.778 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.778+0000 7f9a89381700 1 -- 172.21.15.143:0/202485433 --> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a8410bf70 con 0x7f9a7002ffc0 2024-02-19T21:54:22.910 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.910+0000 7f9a6b7fe700 1 -- 172.21.15.143:0/202485433 <== mgr.14162 v2:172.21.15.143:6800/2497069397 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f9a8410bf70 con 0x7f9a7002ffc0 2024-02-19T21:54:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:21.443080+0000 mgr.smithi143.bissbc (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T21:54:22.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:21.443851+0000 mon.smithi143 (mon.0) 102 : cluster [DBG] mgrmap e15: smithi143.bissbc(active, since 1.05699s) 2024-02-19T21:54:22.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:21.449472+0000 mgr.smithi143.bissbc (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T21:54:22.911 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-19T21:54:22.912 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.912+0000 7f9a89381700 1 -- 172.21.15.143:0/202485433 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9a7002ffc0 msgr2=0x7f9a70032480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:22.912 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.912+0000 7f9a89381700 1 --2- 172.21.15.143:0/202485433 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9a7002ffc0 0x7f9a70032480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9a780096a0 tx=0x7f9a78006b40).stop 2024-02-19T21:54:22.913 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.912+0000 7f9a89381700 1 -- 172.21.15.143:0/202485433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 msgr2=0x7f9a8410ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:22.913 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.912+0000 7f9a89381700 1 --2- 172.21.15.143:0/202485433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a8410ab90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f9a7400bf40 tx=0x7f9a74006ee0).stop 2024-02-19T21:54:22.913 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.912+0000 7f9a89381700 1 -- 172.21.15.143:0/202485433 shutdown_connections 2024-02-19T21:54:22.913 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.912+0000 7f9a89381700 1 --2- 172.21.15.143:0/202485433 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9a7002ffc0 0x7f9a70032480 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:22.913 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.912+0000 7f9a89381700 1 --2- 172.21.15.143:0/202485433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a840fbea0 0x7f9a8410ab90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:22.913 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.912+0000 7f9a89381700 1 -- 172.21.15.143:0/202485433 >> 172.21.15.143:0/202485433 conn(0x7f9a840f76a0 msgr2=0x7f9a840f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:22.913 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.912+0000 7f9a89381700 1 -- 172.21.15.143:0/202485433 shutdown_connections 2024-02-19T21:54:22.913 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:22.912+0000 7f9a89381700 1 -- 172.21.15.143:0/202485433 wait complete. 2024-02-19T21:54:23.546 INFO:teuthology.orchestra.run.smithi143.stderr:Creating initial admin user... 2024-02-19T21:54:23.547 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpugljip8a:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-19T21:54:23.549 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:22.449206+0000 mon.smithi143 (mon.0) 103 : cluster [DBG] mgrmap e16: smithi143.bissbc(active, since 2s) 2024-02-19T21:54:23.549 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:22.780142+0000 mgr.smithi143.bissbc (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:54:23.549 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:22.909289+0000 mon.smithi143 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:23.549 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:22.910991+0000 mon.smithi143 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:24.050 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.048+0000 7f7e9f583700 1 Processor -- start 2024-02-19T21:54:24.050 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.048+0000 7f7e9f583700 1 -- start start 2024-02-19T21:54:24.050 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.049+0000 7f7e9f583700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e980fd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:24.050 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.049+0000 7f7e9f583700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e980fde90 con 0x7f7e980fd580 2024-02-19T21:54:24.051 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.049+0000 7f7e9d31f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e980fd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:24.051 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.049+0000 7f7e9d31f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e980fd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33534/0 (socket says 172.21.15.143:33534) 2024-02-19T21:54:24.051 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.049+0000 7f7e9d31f700 1 -- 172.21.15.143:0/3970886165 learned_addr learned my addr 172.21.15.143:0/3970886165 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:24.051 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.049+0000 7f7e9d31f700 1 -- 172.21.15.143:0/3970886165 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e980fdfd0 con 0x7f7e980fd580 2024-02-19T21:54:24.051 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.049+0000 7f7e9d31f700 1 --2- 172.21.15.143:0/3970886165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e980fd9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f7e8800b5f0 tx=0x7f7e88006ee0).ready entity=mon.0 client_cookie=f6a7a02e136fa670 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:24.051 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.049+0000 7f7e87fff700 1 -- 172.21.15.143:0/3970886165 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e8800d600 con 0x7f7e980fd580 2024-02-19T21:54:24.051 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.049+0000 7f7e87fff700 1 -- 172.21.15.143:0/3970886165 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7e8800dbf0 con 0x7f7e980fd580 2024-02-19T21:54:24.054 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.049+0000 7f7e87fff700 1 -- 172.21.15.143:0/3970886165 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e88011cb0 con 0x7f7e980fd580 2024-02-19T21:54:24.054 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.050+0000 7f7e9f583700 1 -- 172.21.15.143:0/3970886165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 msgr2=0x7f7e980fd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:24.054 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.050+0000 7f7e9f583700 1 --2- 172.21.15.143:0/3970886165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e980fd9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f7e8800b5f0 tx=0x7f7e88006ee0).stop 2024-02-19T21:54:24.054 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.050+0000 7f7e9f583700 1 -- 172.21.15.143:0/3970886165 shutdown_connections 2024-02-19T21:54:24.054 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.050+0000 7f7e9f583700 1 --2- 172.21.15.143:0/3970886165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e980fd9a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:24.054 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.050+0000 7f7e9f583700 1 -- 172.21.15.143:0/3970886165 >> 172.21.15.143:0/3970886165 conn(0x7f7e980f5d20 msgr2=0x7f7e980f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:24.054 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.050+0000 7f7e9f583700 1 -- 172.21.15.143:0/3970886165 shutdown_connections 2024-02-19T21:54:24.054 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.050+0000 7f7e9f583700 1 -- 172.21.15.143:0/3970886165 wait complete. 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.051+0000 7f7e9f583700 1 Processor -- start 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.051+0000 7f7e9f583700 1 -- start start 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.051+0000 7f7e9f583700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e9810aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.051+0000 7f7e9f583700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9810b090 con 0x7f7e980fd580 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.051+0000 7f7e9d31f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e9810aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.051+0000 7f7e9d31f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e9810aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33550/0 (socket says 172.21.15.143:33550) 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.051+0000 7f7e9d31f700 1 -- 172.21.15.143:0/2179566450 learned_addr learned my addr 172.21.15.143:0/2179566450 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.052+0000 7f7e9d31f700 1 -- 172.21.15.143:0/2179566450 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8800b200 con 0x7f7e980fd580 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.052+0000 7f7e9d31f700 1 --2- 172.21.15.143:0/2179566450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e9810aba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f7e8800bf40 tx=0x7f7e88006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.052+0000 7f7e85ffb700 1 -- 172.21.15.143:0/2179566450 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e8801b570 con 0x7f7e980fd580 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.052+0000 7f7e85ffb700 1 -- 172.21.15.143:0/2179566450 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7e8801bb60 con 0x7f7e980fd580 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.052+0000 7f7e9f583700 1 -- 172.21.15.143:0/2179566450 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9810b290 con 0x7f7e980fd580 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.052+0000 7f7e85ffb700 1 -- 172.21.15.143:0/2179566450 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e8801aa20 con 0x7f7e980fd580 2024-02-19T21:54:24.055 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.052+0000 7f7e9f583700 1 -- 172.21.15.143:0/2179566450 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9810b720 con 0x7f7e980fd580 2024-02-19T21:54:24.056 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.053+0000 7f7e85ffb700 1 -- 172.21.15.143:0/2179566450 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f7e8800f070 con 0x7f7e980fd580 2024-02-19T21:54:24.056 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.053+0000 7f7e85ffb700 1 --2- 172.21.15.143:0/2179566450 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7e8002ff70 0x7f7e80032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:24.056 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.053+0000 7f7e85ffb700 1 -- 172.21.15.143:0/2179566450 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e88041760 con 0x7f7e980fd580 2024-02-19T21:54:24.056 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.053+0000 7f7e9f583700 1 -- 172.21.15.143:0/2179566450 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e98045ad0 con 0x7f7e980fd580 2024-02-19T21:54:24.056 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.053+0000 7f7e9cb1e700 1 --2- 172.21.15.143:0/2179566450 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7e8002ff70 0x7f7e80032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:24.056 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.054+0000 7f7e9cb1e700 1 --2- 172.21.15.143:0/2179566450 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7e8002ff70 0x7f7e80032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f7e940096a0 tx=0x7f7e94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:24.058 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.058+0000 7f7e85ffb700 1 -- 172.21.15.143:0/2179566450 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e88016070 con 0x7f7e980fd580 2024-02-19T21:54:24.223 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.222+0000 7f7e9f583700 1 -- 172.21.15.143:0/2179566450 --> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7e980fa940 con 0x7f7e8002ff70 2024-02-19T21:54:24.439 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.438+0000 7f7e85ffb700 1 -- 172.21.15.143:0/2179566450 <== mgr.14162 v2:172.21.15.143:6800/2497069397 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7e980fa940 con 0x7f7e8002ff70 2024-02-19T21:54:24.439 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$cKVC9ooAkO7wlga3.H/fm.zyED5qfT88GN4f.2pVWExdLv4HsYnO.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708379664, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-19T21:54:24.441 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.440+0000 7f7e9f583700 1 -- 172.21.15.143:0/2179566450 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7e8002ff70 msgr2=0x7f7e80032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:24.442 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.440+0000 7f7e9f583700 1 --2- 172.21.15.143:0/2179566450 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7e8002ff70 0x7f7e80032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f7e940096a0 tx=0x7f7e94006b40).stop 2024-02-19T21:54:24.442 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.440+0000 7f7e9f583700 1 -- 172.21.15.143:0/2179566450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 msgr2=0x7f7e9810aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:24.442 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.440+0000 7f7e9f583700 1 --2- 172.21.15.143:0/2179566450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e9810aba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f7e8800bf40 tx=0x7f7e88006ee0).stop 2024-02-19T21:54:24.442 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.440+0000 7f7e9f583700 1 -- 172.21.15.143:0/2179566450 shutdown_connections 2024-02-19T21:54:24.442 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.440+0000 7f7e9f583700 1 --2- 172.21.15.143:0/2179566450 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7e8002ff70 0x7f7e80032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:24.442 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.440+0000 7f7e9f583700 1 --2- 172.21.15.143:0/2179566450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e980fd580 0x7f7e9810aba0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:24.442 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.440+0000 7f7e9f583700 1 -- 172.21.15.143:0/2179566450 >> 172.21.15.143:0/2179566450 conn(0x7f7e980f5d20 msgr2=0x7f7e980ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:24.442 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.441+0000 7f7e9f583700 1 -- 172.21.15.143:0/2179566450 shutdown_connections 2024-02-19T21:54:24.442 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:24.441+0000 7f7e9f583700 1 -- 172.21.15.143:0/2179566450 wait complete. 2024-02-19T21:54:25.071 INFO:teuthology.orchestra.run.smithi143.stderr:Fetching dashboard port number... 2024-02-19T21:54:25.071 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpm89l3603:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2duz2puy:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-19T21:54:25.577 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.575+0000 7f2b0bc53700 1 Processor -- start 2024-02-19T21:54:25.577 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.575+0000 7f2b0bc53700 1 -- start start 2024-02-19T21:54:25.578 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.576+0000 7f2b0bc53700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b04070a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:25.578 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.576+0000 7f2b0bc53700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b040728f0 con 0x7f2b040724d0 2024-02-19T21:54:25.578 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.576+0000 7f2b099ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b04070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:25.578 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.576+0000 7f2b099ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b04070a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33562/0 (socket says 172.21.15.143:33562) 2024-02-19T21:54:25.578 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.576+0000 7f2b099ef700 1 -- 172.21.15.143:0/3416570684 learned_addr learned my addr 172.21.15.143:0/3416570684 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:25.578 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.576+0000 7f2b099ef700 1 -- 172.21.15.143:0/3416570684 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b04070f60 con 0x7f2b040724d0 2024-02-19T21:54:25.578 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.576+0000 7f2b099ef700 1 --2- 172.21.15.143:0/3416570684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b04070a70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2b0000b5f0 tx=0x7f2b00006ee0).ready entity=mon.0 client_cookie=7a2486877f67ff33 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:25.578 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.576+0000 7f2b089ed700 1 -- 172.21.15.143:0/3416570684 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b0000d600 con 0x7f2b040724d0 2024-02-19T21:54:25.578 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.576+0000 7f2b089ed700 1 -- 172.21.15.143:0/3416570684 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2b0000dbf0 con 0x7f2b040724d0 2024-02-19T21:54:25.578 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.576+0000 7f2b089ed700 1 -- 172.21.15.143:0/3416570684 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b00011cb0 con 0x7f2b040724d0 2024-02-19T21:54:25.581 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.577+0000 7f2b0bc53700 1 -- 172.21.15.143:0/3416570684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 msgr2=0x7f2b04070a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:25.581 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.577+0000 7f2b0bc53700 1 --2- 172.21.15.143:0/3416570684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b04070a70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2b0000b5f0 tx=0x7f2b00006ee0).stop 2024-02-19T21:54:25.581 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.577+0000 7f2b0bc53700 1 -- 172.21.15.143:0/3416570684 shutdown_connections 2024-02-19T21:54:25.581 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.577+0000 7f2b0bc53700 1 --2- 172.21.15.143:0/3416570684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b04070a70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.577+0000 7f2b0bc53700 1 -- 172.21.15.143:0/3416570684 >> 172.21.15.143:0/3416570684 conn(0x7f2b040f7640 msgr2=0x7f2b040f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.577+0000 7f2b0bc53700 1 -- 172.21.15.143:0/3416570684 shutdown_connections 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.577+0000 7f2b0bc53700 1 -- 172.21.15.143:0/3416570684 wait complete. 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.578+0000 7f2b0bc53700 1 Processor -- start 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.578+0000 7f2b0bc53700 1 -- start start 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.578+0000 7f2b0bc53700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b0410cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.578+0000 7f2b0bc53700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0410d270 con 0x7f2b040724d0 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.578+0000 7f2b099ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b0410cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.578+0000 7f2b099ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b0410cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33564/0 (socket says 172.21.15.143:33564) 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.578+0000 7f2b099ef700 1 -- 172.21.15.143:0/995366759 learned_addr learned my addr 172.21.15.143:0/995366759 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.579+0000 7f2b099ef700 1 -- 172.21.15.143:0/995366759 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0000b200 con 0x7f2b040724d0 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.579+0000 7f2b099ef700 1 --2- 172.21.15.143:0/995366759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b0410cd80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2b00006800 tx=0x7f2b00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:25.582 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.579+0000 7f2af27fc700 1 -- 172.21.15.143:0/995366759 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b0001b570 con 0x7f2b040724d0 2024-02-19T21:54:25.583 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.579+0000 7f2b0bc53700 1 -- 172.21.15.143:0/995366759 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0410d470 con 0x7f2b040724d0 2024-02-19T21:54:25.583 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.579+0000 7f2af27fc700 1 -- 172.21.15.143:0/995366759 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2b0001bb60 con 0x7f2b040724d0 2024-02-19T21:54:25.583 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.579+0000 7f2af27fc700 1 -- 172.21.15.143:0/995366759 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b0001a9f0 con 0x7f2b040724d0 2024-02-19T21:54:25.583 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.579+0000 7f2b0bc53700 1 -- 172.21.15.143:0/995366759 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0410e0b0 con 0x7f2b040724d0 2024-02-19T21:54:25.583 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.580+0000 7f2af27fc700 1 -- 172.21.15.143:0/995366759 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f2b0000f070 con 0x7f2b040724d0 2024-02-19T21:54:25.583 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.580+0000 7f2af27fc700 1 --2- 172.21.15.143:0/995366759 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2aec02ff70 0x7f2aec032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:25.583 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.580+0000 7f2af27fc700 1 -- 172.21.15.143:0/995366759 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b00041780 con 0x7f2b040724d0 2024-02-19T21:54:25.583 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.580+0000 7f2b091ee700 1 --2- 172.21.15.143:0/995366759 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2aec02ff70 0x7f2aec032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:25.583 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.580+0000 7f2b0bc53700 1 -- 172.21.15.143:0/995366759 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0410d900 con 0x7f2b040724d0 2024-02-19T21:54:25.583 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.581+0000 7f2b091ee700 1 --2- 172.21.15.143:0/995366759 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2aec02ff70 0x7f2aec032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2af40096a0 tx=0x7f2af4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:25.585 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.585+0000 7f2af27fc700 1 -- 172.21.15.143:0/995366759 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b00010360 con 0x7f2b040724d0 2024-02-19T21:54:25.743 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.742+0000 7f2b0bc53700 1 -- 172.21.15.143:0/995366759 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f2b0410eee0 con 0x7f2b040724d0 2024-02-19T21:54:25.743 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.743+0000 7f2af27fc700 1 -- 172.21.15.143:0/995366759 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f2b00022ac0 con 0x7f2b040724d0 2024-02-19T21:54:25.744 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 8443 2024-02-19T21:54:25.746 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.744+0000 7f2b0bc53700 1 -- 172.21.15.143:0/995366759 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2aec02ff70 msgr2=0x7f2aec032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:25.746 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.744+0000 7f2b0bc53700 1 --2- 172.21.15.143:0/995366759 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2aec02ff70 0x7f2aec032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2af40096a0 tx=0x7f2af4006b40).stop 2024-02-19T21:54:25.746 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.744+0000 7f2b0bc53700 1 -- 172.21.15.143:0/995366759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 msgr2=0x7f2b0410cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:25.746 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.744+0000 7f2b0bc53700 1 --2- 172.21.15.143:0/995366759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b0410cd80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2b00006800 tx=0x7f2b00006ee0).stop 2024-02-19T21:54:25.746 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.744+0000 7f2b0bc53700 1 -- 172.21.15.143:0/995366759 shutdown_connections 2024-02-19T21:54:25.746 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.744+0000 7f2b0bc53700 1 --2- 172.21.15.143:0/995366759 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2aec02ff70 0x7f2aec032430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:25.746 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.744+0000 7f2b0bc53700 1 --2- 172.21.15.143:0/995366759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b040724d0 0x7f2b0410cd80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:25.746 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.744+0000 7f2b0bc53700 1 -- 172.21.15.143:0/995366759 >> 172.21.15.143:0/995366759 conn(0x7f2b040f7640 msgr2=0x7f2b040f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:25.746 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.745+0000 7f2b0bc53700 1 -- 172.21.15.143:0/995366759 shutdown_connections 2024-02-19T21:54:25.746 INFO:teuthology.orchestra.run.smithi143.stderr:/usr/bin/ceph: 2024-02-19T21:54:25.745+0000 7f2b0bc53700 1 -- 172.21.15.143:0/995366759 wait complete. 2024-02-19T21:54:25.764 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:25 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:24.224890+0000 mgr.smithi143.bissbc (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:54:25.764 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:25 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:24.439919+0000 mon.smithi143 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:26.370 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T21:54:26.380 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: disabled 2024-02-19T21:54:26.380 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T21:54:26.387 INFO:teuthology.orchestra.run.smithi143.stderr:systemctl: inactive 2024-02-19T21:54:26.388 INFO:teuthology.orchestra.run.smithi143.stderr:firewalld.service is not enabled 2024-02-19T21:54:26.388 INFO:teuthology.orchestra.run.smithi143.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-19T21:54:26.391 INFO:teuthology.orchestra.run.smithi143.stderr:Ceph Dashboard is now available at: 2024-02-19T21:54:26.391 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-02-19T21:54:26.391 INFO:teuthology.orchestra.run.smithi143.stderr: URL: https://smithi143.front.sepia.ceph.com:8443/ 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr: User: admin 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr: Password: h854y76cp1 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr:You can access the Ceph CLI with: 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 38187086-cf71-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr: ceph telemetry on 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr:For more information see: 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-02-19T21:54:26.392 INFO:teuthology.orchestra.run.smithi143.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-19T21:54:26.393 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-02-19T21:54:26.393 INFO:teuthology.orchestra.run.smithi143.stderr:Bootstrap complete. 2024-02-19T21:54:26.393 INFO:teuthology.orchestra.run.smithi143.stderr:Releasing lock 139927742573032 on /run/cephadm/38187086-cf71-11ee-95bb-87774f69a715.lock 2024-02-19T21:54:26.393 INFO:teuthology.orchestra.run.smithi143.stderr:Lock 139927742573032 released on /run/cephadm/38187086-cf71-11ee-95bb-87774f69a715.lock 2024-02-19T21:54:26.424 INFO:tasks.cephadm:Fetching config... 2024-02-19T21:54:26.425 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-02-19T21:54:26.425 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-19T21:54:26.483 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-19T21:54:26.483 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-02-19T21:54:26.483 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-19T21:54:26.541 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-19T21:54:26.541 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-02-19T21:54:26.541 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/mon.smithi143/keyring of=/dev/stdout 2024-02-19T21:54:26.608 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-19T21:54:26.609 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-02-19T21:54:26.609 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-19T21:54:26.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:25.744756+0000 mon.smithi143 (mon.0) 107 : audit [DBG] from='client.? 172.21.15.143:0/995366759' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-19T21:54:26.666 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-19T21:54:26.666 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvuJIirlmehmbHme0GOsCEKht/SPbbEWBAQAooTY3BYoiMyYX3+8HvAiDLl0PNh17Hd6C9TaiNJZs/XiqM+uDdHECEln6feGQ3hfNE9RUGaxSciJVLIrwHEI+RjD+u/Le7qcqFjseH6Zv1jJLJN1CIe86hDYKbvmKYZOiEHI0UZGSeHHNPbnTDUkO7TfefMc2DiaW0qh7RYDUzbvqE9AUqX2f+Ln1vrReh5bAmNmeigfsiTtqSjcC3RGV3BD9QECEnOwLNmVZsQn8B0oqb9sHy1IgVRAspa2P+YoX96ecFhTUE25BhuPPPl+J5FZv/rVAAQmbw30ssXAcKwZKP6jvKVj6leGA+ljzkx/pfKz+95dQgUta/CF5oKbDvGhEYFcNvN1z+9bweBgJP+BMHk3eFvALliud2hFigpuv8hovA8LmeAx/85GZ7YZOsiNEhvcgRDQzxgobZCg+PGgrAam1VGxSVDr+9TbrRb2aKeeRUO5wawg3lrQ9RdQQop262/Nc= ceph-38187086-cf71-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T21:54:26.741 INFO:teuthology.orchestra.run.smithi143.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvuJIirlmehmbHme0GOsCEKht/SPbbEWBAQAooTY3BYoiMyYX3+8HvAiDLl0PNh17Hd6C9TaiNJZs/XiqM+uDdHECEln6feGQ3hfNE9RUGaxSciJVLIrwHEI+RjD+u/Le7qcqFjseH6Zv1jJLJN1CIe86hDYKbvmKYZOiEHI0UZGSeHHNPbnTDUkO7TfefMc2DiaW0qh7RYDUzbvqE9AUqX2f+Ln1vrReh5bAmNmeigfsiTtqSjcC3RGV3BD9QECEnOwLNmVZsQn8B0oqb9sHy1IgVRAspa2P+YoX96ecFhTUE25BhuPPPl+J5FZv/rVAAQmbw30ssXAcKwZKP6jvKVj6leGA+ljzkx/pfKz+95dQgUta/CF5oKbDvGhEYFcNvN1z+9bweBgJP+BMHk3eFvALliud2hFigpuv8hovA8LmeAx/85GZ7YZOsiNEhvcgRDQzxgobZCg+PGgrAam1VGxSVDr+9TbrRb2aKeeRUO5wawg3lrQ9RdQQop262/Nc= ceph-38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:54:26.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvuJIirlmehmbHme0GOsCEKht/SPbbEWBAQAooTY3BYoiMyYX3+8HvAiDLl0PNh17Hd6C9TaiNJZs/XiqM+uDdHECEln6feGQ3hfNE9RUGaxSciJVLIrwHEI+RjD+u/Le7qcqFjseH6Zv1jJLJN1CIe86hDYKbvmKYZOiEHI0UZGSeHHNPbnTDUkO7TfefMc2DiaW0qh7RYDUzbvqE9AUqX2f+Ln1vrReh5bAmNmeigfsiTtqSjcC3RGV3BD9QECEnOwLNmVZsQn8B0oqb9sHy1IgVRAspa2P+YoX96ecFhTUE25BhuPPPl+J5FZv/rVAAQmbw30ssXAcKwZKP6jvKVj6leGA+ljzkx/pfKz+95dQgUta/CF5oKbDvGhEYFcNvN1z+9bweBgJP+BMHk3eFvALliud2hFigpuv8hovA8LmeAx/85GZ7YZOsiNEhvcgRDQzxgobZCg+PGgrAam1VGxSVDr+9TbrRb2aKeeRUO5wawg3lrQ9RdQQop262/Nc= ceph-38187086-cf71-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T21:54:26.798 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvuJIirlmehmbHme0GOsCEKht/SPbbEWBAQAooTY3BYoiMyYX3+8HvAiDLl0PNh17Hd6C9TaiNJZs/XiqM+uDdHECEln6feGQ3hfNE9RUGaxSciJVLIrwHEI+RjD+u/Le7qcqFjseH6Zv1jJLJN1CIe86hDYKbvmKYZOiEHI0UZGSeHHNPbnTDUkO7TfefMc2DiaW0qh7RYDUzbvqE9AUqX2f+Ln1vrReh5bAmNmeigfsiTtqSjcC3RGV3BD9QECEnOwLNmVZsQn8B0oqb9sHy1IgVRAspa2P+YoX96ecFhTUE25BhuPPPl+J5FZv/rVAAQmbw30ssXAcKwZKP6jvKVj6leGA+ljzkx/pfKz+95dQgUta/CF5oKbDvGhEYFcNvN1z+9bweBgJP+BMHk3eFvALliud2hFigpuv8hovA8LmeAx/85GZ7YZOsiNEhvcgRDQzxgobZCg+PGgrAam1VGxSVDr+9TbrRb2aKeeRUO5wawg3lrQ9RdQQop262/Nc= ceph-38187086-cf71-11ee-95bb-87774f69a715 2024-02-19T21:54:26.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-19T21:54:27.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:27 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:26.442143+0000 mon.smithi143 (mon.0) 108 : cluster [DBG] mgrmap e17: smithi143.bissbc(active, since 6s) 2024-02-19T21:54:29.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.859+0000 7f44d55a3700 1 -- 172.21.15.143:0/774030755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d0060200 msgr2=0x7f44d00fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:29.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.859+0000 7f44d55a3700 1 --2- 172.21.15.143:0/774030755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d0060200 0x7f44d00fb570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f44c0009fb0 tx=0x7f44c0009b30).stop 2024-02-19T21:54:29.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.859+0000 7f44d55a3700 1 -- 172.21.15.143:0/774030755 shutdown_connections 2024-02-19T21:54:29.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.859+0000 7f44d55a3700 1 --2- 172.21.15.143:0/774030755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d0060200 0x7f44d00fb570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:29.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.859+0000 7f44d55a3700 1 -- 172.21.15.143:0/774030755 >> 172.21.15.143:0/774030755 conn(0x7f44d00f0b70 msgr2=0x7f44d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:29.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.859+0000 7f44d55a3700 1 -- 172.21.15.143:0/774030755 shutdown_connections 2024-02-19T21:54:29.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.859+0000 7f44d55a3700 1 -- 172.21.15.143:0/774030755 wait complete. 2024-02-19T21:54:29.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.860+0000 7f44d55a3700 1 Processor -- start 2024-02-19T21:54:29.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.860+0000 7f44d55a3700 1 -- start start 2024-02-19T21:54:29.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.860+0000 7f44d55a3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d0060200 0x7f44d0108990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.860+0000 7f44d55a3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d0108e80 con 0x7f44d0060200 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.860+0000 7f44ceffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d0060200 0x7f44d0108990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.860+0000 7f44ceffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d0060200 0x7f44d0108990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:33574/0 (socket says 172.21.15.143:33574) 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.860+0000 7f44ceffd700 1 -- 172.21.15.143:0/891428800 learned_addr learned my addr 172.21.15.143:0/891428800 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.860+0000 7f44ceffd700 1 -- 172.21.15.143:0/891428800 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c0005040 con 0x7f44d0060200 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.860+0000 7f44ceffd700 1 --2- 172.21.15.143:0/891428800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d0060200 0x7f44d0108990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f44c00089f0 tx=0x7f44c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.861+0000 7f44b77fe700 1 -- 172.21.15.143:0/891428800 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44c0009320 con 0x7f44d0060200 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.861+0000 7f44b77fe700 1 -- 172.21.15.143:0/891428800 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f44c00136f0 con 0x7f44d0060200 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.861+0000 7f44d55a3700 1 -- 172.21.15.143:0/891428800 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d0109080 con 0x7f44d0060200 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.861+0000 7f44b77fe700 1 -- 172.21.15.143:0/891428800 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44c0014420 con 0x7f44d0060200 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.861+0000 7f44d55a3700 1 -- 172.21.15.143:0/891428800 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d0109510 con 0x7f44d0060200 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.861+0000 7f44b77fe700 1 -- 172.21.15.143:0/891428800 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44c0014580 con 0x7f44d0060200 2024-02-19T21:54:29.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.861+0000 7f44d55a3700 1 -- 172.21.15.143:0/891428800 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d0045ad0 con 0x7f44d0060200 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.862+0000 7f44b77fe700 1 --2- 172.21.15.143:0/891428800 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f44b802ffb0 0x7f44b8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.862+0000 7f44b77fe700 1 -- 172.21.15.143:0/891428800 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44c0040e50 con 0x7f44d0060200 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.862+0000 7f44ce7fc700 1 --2- 172.21.15.143:0/891428800 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f44b802ffb0 0x7f44b8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.862+0000 7f44ce7fc700 1 --2- 172.21.15.143:0/891428800 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f44b802ffb0 0x7f44b8032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f44c40096a0 tx=0x7f44c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:28.866+0000 7f44b77fe700 1 -- 172.21.15.143:0/891428800 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44c001cd50 con 0x7f44d0060200 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.020+0000 7f44d55a3700 1 -- 172.21.15.143:0/891428800 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f44d010a3d0 con 0x7f44d0060200 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.024+0000 7f44b77fe700 1 -- 172.21.15.143:0/891428800 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f44c001c970 con 0x7f44d0060200 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.026+0000 7f44d55a3700 1 -- 172.21.15.143:0/891428800 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f44b802ffb0 msgr2=0x7f44b8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.026+0000 7f44d55a3700 1 --2- 172.21.15.143:0/891428800 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f44b802ffb0 0x7f44b8032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f44c40096a0 tx=0x7f44c4006b40).stop 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.026+0000 7f44d55a3700 1 -- 172.21.15.143:0/891428800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d0060200 msgr2=0x7f44d0108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.026+0000 7f44d55a3700 1 --2- 172.21.15.143:0/891428800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d0060200 0x7f44d0108990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f44c00089f0 tx=0x7f44c000b040).stop 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.026+0000 7f44d55a3700 1 -- 172.21.15.143:0/891428800 shutdown_connections 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.026+0000 7f44d55a3700 1 --2- 172.21.15.143:0/891428800 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f44b802ffb0 0x7f44b8032470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:29.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.026+0000 7f44d55a3700 1 --2- 172.21.15.143:0/891428800 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d0060200 0x7f44d0108990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:29.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.026+0000 7f44d55a3700 1 -- 172.21.15.143:0/891428800 >> 172.21.15.143:0/891428800 conn(0x7f44d00f0b70 msgr2=0x7f44d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:29.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.026+0000 7f44d55a3700 1 -- 172.21.15.143:0/891428800 shutdown_connections 2024-02-19T21:54:29.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:29.026+0000 7f44d55a3700 1 -- 172.21.15.143:0/891428800 wait complete. 2024-02-19T21:54:29.887 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-02-19T21:54:29.887 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-19T21:54:29.887 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-02-19T21:54:29.909 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-02-19T21:54:29.909 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T21:54:29.971 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-02-19T21:54:29.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph orch host add smithi173 2024-02-19T21:54:30.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:29.025576+0000 mon.smithi143 (mon.0) 109 : audit [INF] from='client.? 172.21.15.143:0/891428800' entity='client.admin' 2024-02-19T21:54:31.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:31 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:30.580775+0000 mon.smithi143 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:31.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:31 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:30.801301+0000 mon.smithi143 (mon.0) 111 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:33.016 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:31.746037+0000 mon.smithi143 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:36.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:34.944715+0000 mon.smithi143 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:36.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:35.210954+0000 mon.smithi143 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:36.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:35.213065+0000 mon.smithi143 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:36.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:54:35.218864+0000 mgr.smithi143.bissbc (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi143 on smithi143 2024-02-19T21:54:41.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:41 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:40.442767+0000 mgr.smithi143.bissbc (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:54:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:42.443103+0000 mgr.smithi143.bissbc (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:54:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:44.443400+0000 mgr.smithi143.bissbc (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:54:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:46.443763+0000 mgr.smithi143.bissbc (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:54:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:48.444083+0000 mgr.smithi143.bissbc (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:54:52.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.431+0000 7fbd6051b700 1 -- 172.21.15.173:0/3036119985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd580f5410 msgr2=0x7fbd580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.431+0000 7fbd6051b700 1 --2- 172.21.15.173:0/3036119985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd580f5410 0x7fbd580f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fbd54009fb0 tx=0x7fbd54009b30).stop 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.431+0000 7fbd6051b700 1 -- 172.21.15.173:0/3036119985 shutdown_connections 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.431+0000 7fbd6051b700 1 --2- 172.21.15.173:0/3036119985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd580f5410 0x7fbd580f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.431+0000 7fbd6051b700 1 -- 172.21.15.173:0/3036119985 >> 172.21.15.173:0/3036119985 conn(0x7fbd580f0b70 msgr2=0x7fbd580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.431+0000 7fbd6051b700 1 -- 172.21.15.173:0/3036119985 shutdown_connections 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.431+0000 7fbd6051b700 1 -- 172.21.15.173:0/3036119985 wait complete. 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.431+0000 7fbd6051b700 1 Processor -- start 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.431+0000 7fbd6051b700 1 -- start start 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd6051b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd580f5410 0x7fbd581067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd6051b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd58106cd0 con 0x7fbd580f5410 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd5e2b7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd580f5410 0x7fbd581067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd5e2b7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd580f5410 0x7fbd581067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43548/0 (socket says 172.21.15.173:43548) 2024-02-19T21:54:52.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd5e2b7700 1 -- 172.21.15.173:0/3052520413 learned_addr learned my addr 172.21.15.173:0/3052520413 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd5e2b7700 1 -- 172.21.15.173:0/3052520413 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd54005040 con 0x7fbd580f5410 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd5e2b7700 1 --2- 172.21.15.173:0/3052520413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd580f5410 0x7fbd581067e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fbd54005010 tx=0x7fbd5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd4effd700 1 -- 172.21.15.173:0/3052520413 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd54009320 con 0x7fbd580f5410 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd6051b700 1 -- 172.21.15.173:0/3052520413 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd58106ed0 con 0x7fbd580f5410 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd4effd700 1 -- 172.21.15.173:0/3052520413 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd54013380 con 0x7fbd580f5410 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd4effd700 1 -- 172.21.15.173:0/3052520413 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd54014e50 con 0x7fbd580f5410 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.432+0000 7fbd6051b700 1 -- 172.21.15.173:0/3052520413 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd58107360 con 0x7fbd580f5410 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.433+0000 7fbd4effd700 1 -- 172.21.15.173:0/3052520413 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd540095f0 con 0x7fbd580f5410 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.433+0000 7fbd4effd700 1 --2- 172.21.15.173:0/3052520413 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbd4402ff10 0x7fbd440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.433+0000 7fbd6051b700 1 -- 172.21.15.173:0/3052520413 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd58045ad0 con 0x7fbd580f5410 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.434+0000 7fbd4effd700 1 -- 172.21.15.173:0/3052520413 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd54040e90 con 0x7fbd580f5410 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.434+0000 7fbd5dab6700 1 --2- 172.21.15.173:0/3052520413 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbd4402ff10 0x7fbd440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:52.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.434+0000 7fbd5dab6700 1 --2- 172.21.15.173:0/3052520413 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbd4402ff10 0x7fbd440323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbd480096a0 tx=0x7fbd48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:52.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.439+0000 7fbd4effd700 1 -- 172.21.15.173:0/3052520413 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd5401dbc0 con 0x7fbd580f5410 2024-02-19T21:54:52.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:52.608+0000 7fbd6051b700 1 -- 172.21.15.173:0/3052520413 --> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd580f1d20 con 0x7fbd4402ff10 2024-02-19T21:54:52.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:50.444293+0000 mgr.smithi143.bissbc (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:54:53.215 INFO:teuthology.orchestra.run.smithi173.stdout:Added host 'smithi173' 2024-02-19T21:54:53.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:53.214+0000 7fbd4effd700 1 -- 172.21.15.173:0/3052520413 <== mgr.14162 v2:172.21.15.143:6800/2497069397 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fbd580f1d20 con 0x7fbd4402ff10 2024-02-19T21:54:53.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:53.217+0000 7fbd6051b700 1 -- 172.21.15.173:0/3052520413 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbd4402ff10 msgr2=0x7fbd440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:53.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:53.217+0000 7fbd6051b700 1 --2- 172.21.15.173:0/3052520413 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbd4402ff10 0x7fbd440323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbd480096a0 tx=0x7fbd48006b40).stop 2024-02-19T21:54:53.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:53.217+0000 7fbd6051b700 1 -- 172.21.15.173:0/3052520413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd580f5410 msgr2=0x7fbd581067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:53.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:53.217+0000 7fbd6051b700 1 --2- 172.21.15.173:0/3052520413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd580f5410 0x7fbd581067e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fbd54005010 tx=0x7fbd5400b040).stop 2024-02-19T21:54:53.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:53.217+0000 7fbd6051b700 1 -- 172.21.15.173:0/3052520413 shutdown_connections 2024-02-19T21:54:53.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:53.217+0000 7fbd6051b700 1 --2- 172.21.15.173:0/3052520413 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbd4402ff10 0x7fbd440323d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:53.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:53.217+0000 7fbd6051b700 1 --2- 172.21.15.173:0/3052520413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd580f5410 0x7fbd581067e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:53.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:53.217+0000 7fbd6051b700 1 -- 172.21.15.173:0/3052520413 >> 172.21.15.173:0/3052520413 conn(0x7fbd580f0b70 msgr2=0x7fbd580f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:53.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:53.217+0000 7fbd6051b700 1 -- 172.21.15.173:0/3052520413 shutdown_connections 2024-02-19T21:54:53.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:53.217+0000 7fbd6051b700 1 -- 172.21.15.173:0/3052520413 wait complete. 2024-02-19T21:54:53.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-19T21:54:54.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:52.444560+0000 mgr.smithi143.bissbc (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:54:54.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:52.610500+0000 mgr.smithi143.bissbc (mgr.14162) 13 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:54:54.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:53.214685+0000 mon.smithi143 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:54:54.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:54:53.215156+0000 mgr.smithi143.bissbc (mgr.14162) 14 : cephadm [INF] Added host smithi173 2024-02-19T21:54:55.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.657+0000 7f07ecb45700 1 -- 172.21.15.173:0/3791503526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07e80601c0 msgr2=0x7f07e80f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:55.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.657+0000 7f07ecb45700 1 --2- 172.21.15.173:0/3791503526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07e80601c0 0x7f07e80f91b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f07d8009fb0 tx=0x7f07d8009b30).stop 2024-02-19T21:54:55.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.657+0000 7f07ecb45700 1 -- 172.21.15.173:0/3791503526 shutdown_connections 2024-02-19T21:54:55.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.657+0000 7f07ecb45700 1 --2- 172.21.15.173:0/3791503526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07e80601c0 0x7f07e80f91b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:55.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.657+0000 7f07ecb45700 1 -- 172.21.15.173:0/3791503526 >> 172.21.15.173:0/3791503526 conn(0x7f07e80f0b70 msgr2=0x7f07e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:55.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.657+0000 7f07ecb45700 1 -- 172.21.15.173:0/3791503526 shutdown_connections 2024-02-19T21:54:55.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.657+0000 7f07ecb45700 1 -- 172.21.15.173:0/3791503526 wait complete. 2024-02-19T21:54:55.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.657+0000 7f07ecb45700 1 Processor -- start 2024-02-19T21:54:55.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.657+0000 7f07ecb45700 1 -- start start 2024-02-19T21:54:55.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.658+0000 7f07ecb45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07e80601c0 0x7f07e81067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:55.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.658+0000 7f07ecb45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e8106cb0 con 0x7f07e80601c0 2024-02-19T21:54:55.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.658+0000 7f07e659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07e80601c0 0x7f07e81067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:55.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.658+0000 7f07e659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07e80601c0 0x7f07e81067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43572/0 (socket says 172.21.15.173:43572) 2024-02-19T21:54:55.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.658+0000 7f07e659c700 1 -- 172.21.15.173:0/2265973204 learned_addr learned my addr 172.21.15.173:0/2265973204 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:54:55.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.658+0000 7f07e659c700 1 -- 172.21.15.173:0/2265973204 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d8005040 con 0x7f07e80601c0 2024-02-19T21:54:55.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.658+0000 7f07e659c700 1 --2- 172.21.15.173:0/2265973204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07e80601c0 0x7f07e81067c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f07d80089f0 tx=0x7f07d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:55.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.659+0000 7f07d6ffd700 1 -- 172.21.15.173:0/2265973204 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07d8009320 con 0x7f07e80601c0 2024-02-19T21:54:55.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.659+0000 7f07ecb45700 1 -- 172.21.15.173:0/2265973204 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e8106eb0 con 0x7f07e80601c0 2024-02-19T21:54:55.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.659+0000 7f07d6ffd700 1 -- 172.21.15.173:0/2265973204 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07d8013380 con 0x7f07e80601c0 2024-02-19T21:54:55.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.659+0000 7f07d6ffd700 1 -- 172.21.15.173:0/2265973204 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07d8014e20 con 0x7f07e80601c0 2024-02-19T21:54:55.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.659+0000 7f07ecb45700 1 -- 172.21.15.173:0/2265973204 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e8107340 con 0x7f07e80601c0 2024-02-19T21:54:55.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.659+0000 7f07d6ffd700 1 -- 172.21.15.173:0/2265973204 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07d80263a0 con 0x7f07e80601c0 2024-02-19T21:54:55.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.660+0000 7f07d6ffd700 1 --2- 172.21.15.173:0/2265973204 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07d002ff10 0x7f07d00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:55.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.660+0000 7f07d6ffd700 1 -- 172.21.15.173:0/2265973204 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07d8040f60 con 0x7f07e80601c0 2024-02-19T21:54:55.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.660+0000 7f07ecb45700 1 -- 172.21.15.173:0/2265973204 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e8045ad0 con 0x7f07e80601c0 2024-02-19T21:54:55.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.660+0000 7f07e5d9b700 1 --2- 172.21.15.173:0/2265973204 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07d002ff10 0x7f07d00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:55.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.660+0000 7f07e5d9b700 1 --2- 172.21.15.173:0/2265973204 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07d002ff10 0x7f07d00323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f07dc0096a0 tx=0x7f07dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:55.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.665+0000 7f07d6ffd700 1 -- 172.21.15.173:0/2265973204 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07d8012920 con 0x7f07e80601c0 2024-02-19T21:54:55.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.836+0000 7f07ecb45700 1 -- 172.21.15.173:0/2265973204 --> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f07e80f1d70 con 0x7f07d002ff10 2024-02-19T21:54:55.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.837+0000 7f07d6ffd700 1 -- 172.21.15.173:0/2265973204 <== mgr.14162 v2:172.21.15.143:6800/2497069397 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f07e80f1d70 con 0x7f07d002ff10 2024-02-19T21:54:55.837 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:54:55.837 INFO:teuthology.orchestra.run.smithi173.stdout:[{"addr": "172.21.15.143", "hostname": "smithi143", "labels": [], "status": ""}, {"addr": "smithi173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-02-19T21:54:55.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.839+0000 7f07ecb45700 1 -- 172.21.15.173:0/2265973204 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07d002ff10 msgr2=0x7f07d00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:55.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.839+0000 7f07ecb45700 1 --2- 172.21.15.173:0/2265973204 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07d002ff10 0x7f07d00323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f07dc0096a0 tx=0x7f07dc006b40).stop 2024-02-19T21:54:55.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.839+0000 7f07ecb45700 1 -- 172.21.15.173:0/2265973204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07e80601c0 msgr2=0x7f07e81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:55.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.839+0000 7f07ecb45700 1 --2- 172.21.15.173:0/2265973204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07e80601c0 0x7f07e81067c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f07d80089f0 tx=0x7f07d800b040).stop 2024-02-19T21:54:55.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.839+0000 7f07ecb45700 1 -- 172.21.15.173:0/2265973204 shutdown_connections 2024-02-19T21:54:55.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.839+0000 7f07ecb45700 1 --2- 172.21.15.173:0/2265973204 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07d002ff10 0x7f07d00323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:55.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.839+0000 7f07ecb45700 1 --2- 172.21.15.173:0/2265973204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07e80601c0 0x7f07e81067c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:55.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.839+0000 7f07ecb45700 1 -- 172.21.15.173:0/2265973204 >> 172.21.15.173:0/2265973204 conn(0x7f07e80f0b70 msgr2=0x7f07e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:55.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.839+0000 7f07ecb45700 1 -- 172.21.15.173:0/2265973204 shutdown_connections 2024-02-19T21:54:55.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:54:55.839+0000 7f07ecb45700 1 -- 172.21.15.173:0/2265973204 wait complete. 2024-02-19T21:54:56.466 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-19T21:54:56.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-19T21:54:56.500 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:54.444926+0000 mgr.smithi143.bissbc (mgr.14162) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:54:57.523 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:57 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:55.838089+0000 mgr.smithi143.bissbc (mgr.14162) 16 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:54:58.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.308+0000 7f86c359e700 1 -- 172.21.15.143:0/3420807513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86c40f51a0 msgr2=0x7f86c40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:58.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.308+0000 7f86c359e700 1 --2- 172.21.15.143:0/3420807513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86c40f51a0 0x7f86c40f55c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f86ac009fb0 tx=0x7f86ac009b30).stop 2024-02-19T21:54:58.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.308+0000 7f86c359e700 1 -- 172.21.15.143:0/3420807513 shutdown_connections 2024-02-19T21:54:58.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.308+0000 7f86c359e700 1 --2- 172.21.15.143:0/3420807513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86c40f51a0 0x7f86c40f55c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:58.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.308+0000 7f86c359e700 1 -- 172.21.15.143:0/3420807513 >> 172.21.15.143:0/3420807513 conn(0x7f86c40f09a0 msgr2=0x7f86c40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:58.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.308+0000 7f86c359e700 1 -- 172.21.15.143:0/3420807513 shutdown_connections 2024-02-19T21:54:58.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.308+0000 7f86c359e700 1 -- 172.21.15.143:0/3420807513 wait complete. 2024-02-19T21:54:58.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.309+0000 7f86c359e700 1 Processor -- start 2024-02-19T21:54:58.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.309+0000 7f86c359e700 1 -- start start 2024-02-19T21:54:58.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.309+0000 7f86c359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86c40f51a0 0x7f86c40fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:58.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.309+0000 7f86c359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40fb6f0 con 0x7f86c40f51a0 2024-02-19T21:54:58.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.309+0000 7f86c259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86c40f51a0 0x7f86c40fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:58.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.309+0000 7f86c259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86c40f51a0 0x7f86c40fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:42360/0 (socket says 172.21.15.143:42360) 2024-02-19T21:54:58.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.309+0000 7f86c259c700 1 -- 172.21.15.143:0/197849331 learned_addr learned my addr 172.21.15.143:0/197849331 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-02-19T21:54:58.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.309+0000 7f86c259c700 1 -- 172.21.15.143:0/197849331 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac005040 con 0x7f86c40f51a0 2024-02-19T21:54:58.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.309+0000 7f86c259c700 1 --2- 172.21.15.143:0/197849331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86c40f51a0 0x7f86c40fd270 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f86ac0089f0 tx=0x7f86ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:58.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.310+0000 7f86baffd700 1 -- 172.21.15.143:0/197849331 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86ac009320 con 0x7f86c40f51a0 2024-02-19T21:54:58.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.310+0000 7f86c359e700 1 -- 172.21.15.143:0/197849331 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c40fb890 con 0x7f86c40f51a0 2024-02-19T21:54:58.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.310+0000 7f86baffd700 1 -- 172.21.15.143:0/197849331 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86ac013380 con 0x7f86c40f51a0 2024-02-19T21:54:58.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.310+0000 7f86baffd700 1 -- 172.21.15.143:0/197849331 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86ac014e20 con 0x7f86c40f51a0 2024-02-19T21:54:58.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.310+0000 7f86c359e700 1 -- 172.21.15.143:0/197849331 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c40fbd20 con 0x7f86c40f51a0 2024-02-19T21:54:58.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.310+0000 7f86baffd700 1 -- 172.21.15.143:0/197849331 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86ac0095f0 con 0x7f86c40f51a0 2024-02-19T21:54:58.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.310+0000 7f86baffd700 1 --2- 172.21.15.143:0/197849331 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f86b002ff10 0x7f86b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:54:58.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.311+0000 7f86baffd700 1 -- 172.21.15.143:0/197849331 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86ac040e90 con 0x7f86c40f51a0 2024-02-19T21:54:58.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.311+0000 7f86c359e700 1 -- 172.21.15.143:0/197849331 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c4059100 con 0x7f86c40f51a0 2024-02-19T21:54:58.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.311+0000 7f86c1d9b700 1 --2- 172.21.15.143:0/197849331 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f86b002ff10 0x7f86b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:54:58.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.311+0000 7f86c1d9b700 1 --2- 172.21.15.143:0/197849331 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f86b002ff10 0x7f86b00323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f86b40096a0 tx=0x7f86b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:54:58.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.315+0000 7f86baffd700 1 -- 172.21.15.143:0/197849331 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86ac014420 con 0x7f86c40f51a0 2024-02-19T21:54:58.472 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:58.473+0000 7f86c359e700 1 -- 172.21.15.143:0/197849331 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f86c40fad30 con 0x7f86c40f51a0 2024-02-19T21:54:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:56.445209+0000 mgr.smithi143.bissbc (mgr.14162) 17 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:54:59.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:59.434+0000 7f86baffd700 1 -- 172.21.15.143:0/197849331 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f86ac01d980 con 0x7f86c40f51a0 2024-02-19T21:54:59.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:59.436+0000 7f86c359e700 1 -- 172.21.15.143:0/197849331 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f86b002ff10 msgr2=0x7f86b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:59.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:59.436+0000 7f86c359e700 1 --2- 172.21.15.143:0/197849331 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f86b002ff10 0x7f86b00323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f86b40096a0 tx=0x7f86b4006b40).stop 2024-02-19T21:54:59.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:59.436+0000 7f86c359e700 1 -- 172.21.15.143:0/197849331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86c40f51a0 msgr2=0x7f86c40fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:54:59.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:59.436+0000 7f86c359e700 1 --2- 172.21.15.143:0/197849331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86c40f51a0 0x7f86c40fd270 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f86ac0089f0 tx=0x7f86ac00b040).stop 2024-02-19T21:54:59.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:59.437+0000 7f86c359e700 1 -- 172.21.15.143:0/197849331 shutdown_connections 2024-02-19T21:54:59.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:59.437+0000 7f86c359e700 1 --2- 172.21.15.143:0/197849331 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f86b002ff10 0x7f86b00323d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:59.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:59.437+0000 7f86c359e700 1 --2- 172.21.15.143:0/197849331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86c40f51a0 0x7f86c40fd270 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:54:59.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:59.437+0000 7f86c359e700 1 -- 172.21.15.143:0/197849331 >> 172.21.15.143:0/197849331 conn(0x7f86c40f09a0 msgr2=0x7f86c405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:54:59.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:59.437+0000 7f86c359e700 1 -- 172.21.15.143:0/197849331 shutdown_connections 2024-02-19T21:54:59.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-02-19T21:54:59.437+0000 7f86c359e700 1 -- 172.21.15.143:0/197849331 wait complete. 2024-02-19T21:54:59.437 INFO:teuthology.orchestra.run.smithi143.stderr:adjusted tunables profile to default 2024-02-19T21:54:59.804 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:54:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:58.473942+0000 mon.smithi143 (mon.0) 117 : audit [INF] from='client.? 172.21.15.143:0/197849331' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-19T21:55:00.143 INFO:tasks.cephadm:Adding mon.smithi143 on smithi143 2024-02-19T21:55:00.143 INFO:tasks.cephadm:Adding mon.smithi173 on smithi173 2024-02-19T21:55:00.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi143:172.21.15.143=smithi143;smithi173:172.21.15.173=smithi173' 2024-02-19T21:55:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:58.445439+0000 mgr.smithi143.bissbc (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:54:59.435122+0000 mon.smithi143 (mon.0) 118 : audit [INF] from='client.? 172.21.15.143:0/197849331' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-19T21:55:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:54:59.435189+0000 mon.smithi143 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-19T21:55:02.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.024+0000 7f46c6b86700 1 -- 172.21.15.173:0/312970798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46c00f5410 msgr2=0x7f46c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:02.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.024+0000 7f46c6b86700 1 --2- 172.21.15.173:0/312970798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46c00f5410 0x7f46c00f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f46b4009fb0 tx=0x7f46b4009b30).stop 2024-02-19T21:55:02.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.024+0000 7f46c6b86700 1 -- 172.21.15.173:0/312970798 shutdown_connections 2024-02-19T21:55:02.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.024+0000 7f46c6b86700 1 --2- 172.21.15.173:0/312970798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46c00f5410 0x7f46c00f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:02.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.024+0000 7f46c6b86700 1 -- 172.21.15.173:0/312970798 >> 172.21.15.173:0/312970798 conn(0x7f46c00f0b70 msgr2=0x7f46c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:02.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.024+0000 7f46c6b86700 1 -- 172.21.15.173:0/312970798 shutdown_connections 2024-02-19T21:55:02.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.024+0000 7f46c6b86700 1 -- 172.21.15.173:0/312970798 wait complete. 2024-02-19T21:55:02.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.024+0000 7f46c6b86700 1 Processor -- start 2024-02-19T21:55:02.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.024+0000 7f46c6b86700 1 -- start start 2024-02-19T21:55:02.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.024+0000 7f46c6b86700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46c00f5410 0x7f46c0105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:02.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.024+0000 7f46c6b86700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c0105c00 con 0x7f46c00f5410 2024-02-19T21:55:02.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.025+0000 7f46c4922700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46c00f5410 0x7f46c0105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:02.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.025+0000 7f46c4922700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46c00f5410 0x7f46c0105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:56524/0 (socket says 172.21.15.173:56524) 2024-02-19T21:55:02.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.025+0000 7f46c4922700 1 -- 172.21.15.173:0/224140267 learned_addr learned my addr 172.21.15.173:0/224140267 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:02.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.025+0000 7f46c4922700 1 -- 172.21.15.173:0/224140267 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b4005040 con 0x7f46c00f5410 2024-02-19T21:55:02.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.025+0000 7f46c4922700 1 --2- 172.21.15.173:0/224140267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46c00f5410 0x7f46c0105710 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f46b4005010 tx=0x7f46b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:02.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.025+0000 7f46bd7fa700 1 -- 172.21.15.173:0/224140267 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46b4009320 con 0x7f46c00f5410 2024-02-19T21:55:02.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.026+0000 7f46c6b86700 1 -- 172.21.15.173:0/224140267 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c0105e00 con 0x7f46c00f5410 2024-02-19T21:55:02.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.026+0000 7f46bd7fa700 1 -- 172.21.15.173:0/224140267 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46b4013380 con 0x7f46c00f5410 2024-02-19T21:55:02.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.026+0000 7f46bd7fa700 1 -- 172.21.15.173:0/224140267 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46b4014e50 con 0x7f46c00f5410 2024-02-19T21:55:02.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.026+0000 7f46c6b86700 1 -- 172.21.15.173:0/224140267 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c0106290 con 0x7f46c00f5410 2024-02-19T21:55:02.028 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.026+0000 7f46bd7fa700 1 -- 172.21.15.173:0/224140267 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46b40095f0 con 0x7f46c00f5410 2024-02-19T21:55:02.028 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.026+0000 7f46c6b86700 1 -- 172.21.15.173:0/224140267 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c0045ad0 con 0x7f46c00f5410 2024-02-19T21:55:02.028 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.027+0000 7f46bd7fa700 1 --2- 172.21.15.173:0/224140267 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46b002ff10 0x7f46b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:02.028 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.027+0000 7f46bd7fa700 1 -- 172.21.15.173:0/224140267 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46b4040e90 con 0x7f46c00f5410 2024-02-19T21:55:02.028 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.027+0000 7f46bffff700 1 --2- 172.21.15.173:0/224140267 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46b002ff10 0x7f46b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:02.028 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.027+0000 7f46bffff700 1 --2- 172.21.15.173:0/224140267 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46b002ff10 0x7f46b00323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f46ac0096a0 tx=0x7f46ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:02.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.032+0000 7f46bd7fa700 1 -- 172.21.15.173:0/224140267 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46b4014420 con 0x7f46c00f5410 2024-02-19T21:55:02.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.207+0000 7f46c6b86700 1 -- 172.21.15.173:0/224140267 --> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi143:172.21.15.143=smithi143;smithi173:172.21.15.173=smithi173", "target": ["mon-mgr", ""]}) v1 -- 0x7f46c00f1da0 con 0x7f46b002ff10 2024-02-19T21:55:02.213 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.213+0000 7f46bd7fa700 1 -- 172.21.15.173:0/224140267 <== mgr.14162 v2:172.21.15.143:6800/2497069397 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f46c00f1da0 con 0x7f46b002ff10 2024-02-19T21:55:02.213 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-02-19T21:55:02.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.215+0000 7f46c6b86700 1 -- 172.21.15.173:0/224140267 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46b002ff10 msgr2=0x7f46b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:02.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.215+0000 7f46c6b86700 1 --2- 172.21.15.173:0/224140267 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46b002ff10 0x7f46b00323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f46ac0096a0 tx=0x7f46ac006b40).stop 2024-02-19T21:55:02.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.215+0000 7f46c6b86700 1 -- 172.21.15.173:0/224140267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46c00f5410 msgr2=0x7f46c0105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:02.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.215+0000 7f46c6b86700 1 --2- 172.21.15.173:0/224140267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46c00f5410 0x7f46c0105710 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f46b4005010 tx=0x7f46b400b040).stop 2024-02-19T21:55:02.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.215+0000 7f46c6b86700 1 -- 172.21.15.173:0/224140267 shutdown_connections 2024-02-19T21:55:02.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.215+0000 7f46c6b86700 1 --2- 172.21.15.173:0/224140267 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46b002ff10 0x7f46b00323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:02.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.215+0000 7f46c6b86700 1 --2- 172.21.15.173:0/224140267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46c00f5410 0x7f46c0105710 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:02.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.215+0000 7f46c6b86700 1 -- 172.21.15.173:0/224140267 >> 172.21.15.173:0/224140267 conn(0x7f46c00f0b70 msgr2=0x7f46c00f15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:02.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.215+0000 7f46c6b86700 1 -- 172.21.15.173:0/224140267 shutdown_connections 2024-02-19T21:55:02.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:02.215+0000 7f46c6b86700 1 -- 172.21.15.173:0/224140267 wait complete. 2024-02-19T21:55:02.886 DEBUG:teuthology.orchestra.run.smithi173:mon.smithi173> sudo journalctl -f -n 0 -u ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi173.service 2024-02-19T21:55:02.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:02.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:00.445679+0000 mgr.smithi143.bissbc (mgr.14162) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:02.213567+0000 mon.smithi143 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:55:02.918 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:-- Logs begin at Mon 2024-02-19 21:38:22 UTC. -- 2024-02-19T21:55:03.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:03 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:02.209482+0000 mgr.smithi143.bissbc (mgr.14162) 20 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi143:172.21.15.143=smithi143;smithi173:172.21.15.173=smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:55:03.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:03 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:55:02.210903+0000 mgr.smithi143.bissbc (mgr.14162) 21 : cephadm [INF] Saving service mon spec with placement smithi143:172.21.15.143=smithi143;smithi173:172.21.15.173=smithi173;count:2 2024-02-19T21:55:04.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.742+0000 7f89b1ec0700 1 -- 172.21.15.173:0/2874826008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac0f5410 msgr2=0x7f89ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:04.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.742+0000 7f89b1ec0700 1 --2- 172.21.15.173:0/2874826008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac0f5410 0x7f89ac0f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f89a0009fb0 tx=0x7f89a0009b30).stop 2024-02-19T21:55:04.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.742+0000 7f89b1ec0700 1 -- 172.21.15.173:0/2874826008 shutdown_connections 2024-02-19T21:55:04.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.742+0000 7f89b1ec0700 1 --2- 172.21.15.173:0/2874826008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac0f5410 0x7f89ac0f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:04.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.742+0000 7f89b1ec0700 1 -- 172.21.15.173:0/2874826008 >> 172.21.15.173:0/2874826008 conn(0x7f89ac0f0b70 msgr2=0x7f89ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:04.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.742+0000 7f89b1ec0700 1 -- 172.21.15.173:0/2874826008 shutdown_connections 2024-02-19T21:55:04.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.742+0000 7f89b1ec0700 1 -- 172.21.15.173:0/2874826008 wait complete. 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.743+0000 7f89b1ec0700 1 Processor -- start 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.743+0000 7f89b1ec0700 1 -- start start 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.743+0000 7f89b1ec0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac0f5410 0x7f89ac1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.743+0000 7f89b1ec0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac106cd0 con 0x7f89ac0f5410 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.743+0000 7f89ab7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac0f5410 0x7f89ac1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.743+0000 7f89ab7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac0f5410 0x7f89ac1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:56528/0 (socket says 172.21.15.173:56528) 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.743+0000 7f89ab7fe700 1 -- 172.21.15.173:0/1637854016 learned_addr learned my addr 172.21.15.173:0/1637854016 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.744+0000 7f89ab7fe700 1 -- 172.21.15.173:0/1637854016 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a0005040 con 0x7f89ac0f5410 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.744+0000 7f89ab7fe700 1 --2- 172.21.15.173:0/1637854016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac0f5410 0x7f89ac1067e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f89a0005010 tx=0x7f89a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.744+0000 7f8993fff700 1 -- 172.21.15.173:0/1637854016 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89a0009320 con 0x7f89ac0f5410 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.744+0000 7f8993fff700 1 -- 172.21.15.173:0/1637854016 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89a0013380 con 0x7f89ac0f5410 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.744+0000 7f89b1ec0700 1 -- 172.21.15.173:0/1637854016 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac106ed0 con 0x7f89ac0f5410 2024-02-19T21:55:04.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.744+0000 7f8993fff700 1 -- 172.21.15.173:0/1637854016 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89a0014e50 con 0x7f89ac0f5410 2024-02-19T21:55:04.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.744+0000 7f89b1ec0700 1 -- 172.21.15.173:0/1637854016 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac107360 con 0x7f89ac0f5410 2024-02-19T21:55:04.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.745+0000 7f8993fff700 1 -- 172.21.15.173:0/1637854016 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89a00095f0 con 0x7f89ac0f5410 2024-02-19T21:55:04.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.745+0000 7f89b1ec0700 1 -- 172.21.15.173:0/1637854016 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac045ad0 con 0x7f89ac0f5410 2024-02-19T21:55:04.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.745+0000 7f8993fff700 1 --2- 172.21.15.173:0/1637854016 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f899402ff10 0x7f89940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:04.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.745+0000 7f8993fff700 1 -- 172.21.15.173:0/1637854016 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89a0040e90 con 0x7f89ac0f5410 2024-02-19T21:55:04.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.745+0000 7f89aaffd700 1 --2- 172.21.15.173:0/1637854016 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f899402ff10 0x7f89940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:04.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.746+0000 7f89aaffd700 1 --2- 172.21.15.173:0/1637854016 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f899402ff10 0x7f89940323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f899c0096a0 tx=0x7f899c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:04.750 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.750+0000 7f8993fff700 1 -- 172.21.15.173:0/1637854016 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89a0014420 con 0x7f89ac0f5410 2024-02-19T21:55:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:02.445912+0000 mgr.smithi143.bissbc (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:04.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.956+0000 7f89b1ec0700 1 -- 172.21.15.173:0/1637854016 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89ac108210 con 0x7f89ac0f5410 2024-02-19T21:55:04.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.957+0000 7f8993fff700 1 -- 172.21.15.173:0/1637854016 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f89a0010230 con 0x7f89ac0f5410 2024-02-19T21:55:04.957 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:04.957 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:04.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.959+0000 7f89b1ec0700 1 -- 172.21.15.173:0/1637854016 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f899402ff10 msgr2=0x7f89940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:04.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.959+0000 7f89b1ec0700 1 --2- 172.21.15.173:0/1637854016 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f899402ff10 0x7f89940323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f899c0096a0 tx=0x7f899c006b40).stop 2024-02-19T21:55:04.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.959+0000 7f89b1ec0700 1 -- 172.21.15.173:0/1637854016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac0f5410 msgr2=0x7f89ac1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:04.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.959+0000 7f89b1ec0700 1 --2- 172.21.15.173:0/1637854016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac0f5410 0x7f89ac1067e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f89a0005010 tx=0x7f89a000b040).stop 2024-02-19T21:55:04.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.959+0000 7f89b1ec0700 1 -- 172.21.15.173:0/1637854016 shutdown_connections 2024-02-19T21:55:04.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.959+0000 7f89b1ec0700 1 --2- 172.21.15.173:0/1637854016 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f899402ff10 0x7f89940323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:04.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.959+0000 7f89b1ec0700 1 --2- 172.21.15.173:0/1637854016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89ac0f5410 0x7f89ac1067e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:04.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.959+0000 7f89b1ec0700 1 -- 172.21.15.173:0/1637854016 >> 172.21.15.173:0/1637854016 conn(0x7f89ac0f0b70 msgr2=0x7f89ac0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:04.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.959+0000 7f89b1ec0700 1 -- 172.21.15.173:0/1637854016 shutdown_connections 2024-02-19T21:55:04.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:04.959+0000 7f89b1ec0700 1 -- 172.21.15.173:0/1637854016 wait complete. 2024-02-19T21:55:04.959 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:05.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:05 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:04.957964+0000 mon.smithi143 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.173:0/1637854016' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:06.646 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:06.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:04.446196+0000 mgr.smithi143.bissbc (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:08.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.734+0000 7fe2802f6700 1 -- 172.21.15.173:0/2452107927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2780f5410 msgr2=0x7fe2780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:08.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.734+0000 7fe2802f6700 1 --2- 172.21.15.173:0/2452107927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2780f5410 0x7fe2780f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fe26c009fb0 tx=0x7fe26c009b30).stop 2024-02-19T21:55:08.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.734+0000 7fe2802f6700 1 -- 172.21.15.173:0/2452107927 shutdown_connections 2024-02-19T21:55:08.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.734+0000 7fe2802f6700 1 --2- 172.21.15.173:0/2452107927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2780f5410 0x7fe2780f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:08.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.734+0000 7fe2802f6700 1 -- 172.21.15.173:0/2452107927 >> 172.21.15.173:0/2452107927 conn(0x7fe2780f0b70 msgr2=0x7fe2780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:08.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.734+0000 7fe2802f6700 1 -- 172.21.15.173:0/2452107927 shutdown_connections 2024-02-19T21:55:08.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.734+0000 7fe2802f6700 1 -- 172.21.15.173:0/2452107927 wait complete. 2024-02-19T21:55:08.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.734+0000 7fe2802f6700 1 Processor -- start 2024-02-19T21:55:08.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.734+0000 7fe2802f6700 1 -- start start 2024-02-19T21:55:08.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe2802f6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2780f5410 0x7fe278109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:08.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe2802f6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe278060220 con 0x7fe2780f5410 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe27e092700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2780f5410 0x7fe278109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe27e092700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2780f5410 0x7fe278109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:56538/0 (socket says 172.21.15.173:56538) 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe27e092700 1 -- 172.21.15.173:0/3533748326 learned_addr learned my addr 172.21.15.173:0/3533748326 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe27e092700 1 -- 172.21.15.173:0/3533748326 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c005040 con 0x7fe2780f5410 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe27e092700 1 --2- 172.21.15.173:0/3533748326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2780f5410 0x7fe278109500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe26c005010 tx=0x7fe26c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe26affd700 1 -- 172.21.15.173:0/3533748326 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe26c009320 con 0x7fe2780f5410 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe2802f6700 1 -- 172.21.15.173:0/3533748326 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe278060420 con 0x7fe2780f5410 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe26affd700 1 -- 172.21.15.173:0/3533748326 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe26c013380 con 0x7fe2780f5410 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe26affd700 1 -- 172.21.15.173:0/3533748326 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe26c014e50 con 0x7fe2780f5410 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.735+0000 7fe2802f6700 1 -- 172.21.15.173:0/3533748326 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe278109bb0 con 0x7fe2780f5410 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.736+0000 7fe26affd700 1 -- 172.21.15.173:0/3533748326 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe26c0095f0 con 0x7fe2780f5410 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.736+0000 7fe2802f6700 1 -- 172.21.15.173:0/3533748326 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe278059100 con 0x7fe2780f5410 2024-02-19T21:55:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.736+0000 7fe26affd700 1 --2- 172.21.15.173:0/3533748326 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe26402ff10 0x7fe2640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:08.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.736+0000 7fe26affd700 1 -- 172.21.15.173:0/3533748326 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe26c040e90 con 0x7fe2780f5410 2024-02-19T21:55:08.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.737+0000 7fe27d891700 1 --2- 172.21.15.173:0/3533748326 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe26402ff10 0x7fe2640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:08.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.737+0000 7fe27d891700 1 --2- 172.21.15.173:0/3533748326 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe26402ff10 0x7fe2640323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe2740096a0 tx=0x7fe274006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:08.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.741+0000 7fe26affd700 1 -- 172.21.15.173:0/3533748326 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe26c014420 con 0x7fe2780f5410 2024-02-19T21:55:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:06.446436+0000 mgr.smithi143.bissbc (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:08.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.953+0000 7fe2802f6700 1 -- 172.21.15.173:0/3533748326 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe278045ad0 con 0x7fe2780f5410 2024-02-19T21:55:08.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.953+0000 7fe26affd700 1 -- 172.21.15.173:0/3533748326 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe26c010230 con 0x7fe2780f5410 2024-02-19T21:55:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:08.954 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:08.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.955+0000 7fe2802f6700 1 -- 172.21.15.173:0/3533748326 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe26402ff10 msgr2=0x7fe2640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:08.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.955+0000 7fe2802f6700 1 --2- 172.21.15.173:0/3533748326 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe26402ff10 0x7fe2640323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe2740096a0 tx=0x7fe274006b40).stop 2024-02-19T21:55:08.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.955+0000 7fe2802f6700 1 -- 172.21.15.173:0/3533748326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2780f5410 msgr2=0x7fe278109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:08.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.955+0000 7fe2802f6700 1 --2- 172.21.15.173:0/3533748326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2780f5410 0x7fe278109500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe26c005010 tx=0x7fe26c00b040).stop 2024-02-19T21:55:08.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.955+0000 7fe2802f6700 1 -- 172.21.15.173:0/3533748326 shutdown_connections 2024-02-19T21:55:08.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.955+0000 7fe2802f6700 1 --2- 172.21.15.173:0/3533748326 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe26402ff10 0x7fe2640323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:08.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.955+0000 7fe2802f6700 1 --2- 172.21.15.173:0/3533748326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2780f5410 0x7fe278109500 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:08.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.955+0000 7fe2802f6700 1 -- 172.21.15.173:0/3533748326 >> 172.21.15.173:0/3533748326 conn(0x7fe2780f0b70 msgr2=0x7fe2780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:08.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.955+0000 7fe2802f6700 1 -- 172.21.15.173:0/3533748326 shutdown_connections 2024-02-19T21:55:08.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:08.955+0000 7fe2802f6700 1 -- 172.21.15.173:0/3533748326 wait complete. 2024-02-19T21:55:08.956 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:09.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:09 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:08.954586+0000 mon.smithi143 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.173:0/3533748326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:10.623 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:10.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:08.446671+0000 mgr.smithi143.bissbc (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:11.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:11 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:10.446952+0000 mgr.smithi143.bissbc (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:12.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.356+0000 7f5c8559a700 1 -- 172.21.15.173:0/1916343555 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c78013990 con 0x7f5c880f51a0 2024-02-19T21:55:12.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.356+0000 7f5c8759e700 1 -- 172.21.15.173:0/1916343555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c880f51a0 msgr2=0x7f5c880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:12.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.356+0000 7f5c8759e700 1 --2- 172.21.15.173:0/1916343555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c880f51a0 0x7f5c880f55c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5c78009fb0 tx=0x7f5c78009b30).stop 2024-02-19T21:55:12.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.356+0000 7f5c8759e700 1 -- 172.21.15.173:0/1916343555 shutdown_connections 2024-02-19T21:55:12.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.356+0000 7f5c8759e700 1 --2- 172.21.15.173:0/1916343555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c880f51a0 0x7f5c880f55c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:12.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.356+0000 7f5c8759e700 1 -- 172.21.15.173:0/1916343555 >> 172.21.15.173:0/1916343555 conn(0x7f5c880f09a0 msgr2=0x7f5c880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:12.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.356+0000 7f5c8759e700 1 -- 172.21.15.173:0/1916343555 shutdown_connections 2024-02-19T21:55:12.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.356+0000 7f5c8759e700 1 -- 172.21.15.173:0/1916343555 wait complete. 2024-02-19T21:55:12.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.357+0000 7f5c8759e700 1 Processor -- start 2024-02-19T21:55:12.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.357+0000 7f5c8759e700 1 -- start start 2024-02-19T21:55:12.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.357+0000 7f5c8759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c880f51a0 0x7f5c881087a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:12.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.357+0000 7f5c8759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c88108c90 con 0x7f5c880f51a0 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.357+0000 7f5c8659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c880f51a0 0x7f5c881087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.357+0000 7f5c8659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c880f51a0 0x7f5c881087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:39212/0 (socket says 172.21.15.173:39212) 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.357+0000 7f5c8659c700 1 -- 172.21.15.173:0/1671855630 learned_addr learned my addr 172.21.15.173:0/1671855630 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.357+0000 7f5c8659c700 1 -- 172.21.15.173:0/1671855630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c78005040 con 0x7f5c880f51a0 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.357+0000 7f5c8659c700 1 --2- 172.21.15.173:0/1671855630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c880f51a0 0x7f5c881087a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f5c780089f0 tx=0x7f5c7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.358+0000 7f5c76ffd700 1 -- 172.21.15.173:0/1671855630 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c78009320 con 0x7f5c880f51a0 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.358+0000 7f5c8759e700 1 -- 172.21.15.173:0/1671855630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c88108e90 con 0x7f5c880f51a0 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.358+0000 7f5c76ffd700 1 -- 172.21.15.173:0/1671855630 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c78013380 con 0x7f5c880f51a0 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.358+0000 7f5c76ffd700 1 -- 172.21.15.173:0/1671855630 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c78014e20 con 0x7f5c880f51a0 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.358+0000 7f5c8759e700 1 -- 172.21.15.173:0/1671855630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c88109320 con 0x7f5c880f51a0 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.358+0000 7f5c76ffd700 1 -- 172.21.15.173:0/1671855630 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c78014420 con 0x7f5c880f51a0 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.358+0000 7f5c8759e700 1 -- 172.21.15.173:0/1671855630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c88045ad0 con 0x7f5c880f51a0 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.359+0000 7f5c76ffd700 1 --2- 172.21.15.173:0/1671855630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5c7002ff10 0x7f5c700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:12.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.359+0000 7f5c76ffd700 1 -- 172.21.15.173:0/1671855630 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c78040fd0 con 0x7f5c880f51a0 2024-02-19T21:55:12.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.359+0000 7f5c85d9b700 1 --2- 172.21.15.173:0/1671855630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5c7002ff10 0x7f5c700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:12.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.359+0000 7f5c85d9b700 1 --2- 172.21.15.173:0/1671855630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5c7002ff10 0x7f5c700323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f5c7c0096a0 tx=0x7f5c7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:12.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.363+0000 7f5c76ffd700 1 -- 172.21.15.173:0/1671855630 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c780095f0 con 0x7f5c880f51a0 2024-02-19T21:55:12.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.569+0000 7f5c8759e700 1 -- 172.21.15.173:0/1671855630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c880f9c50 con 0x7f5c880f51a0 2024-02-19T21:55:12.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.570+0000 7f5c76ffd700 1 -- 172.21.15.173:0/1671855630 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5c7801db70 con 0x7f5c880f51a0 2024-02-19T21:55:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:12.570 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:12.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.571+0000 7f5c8759e700 1 -- 172.21.15.173:0/1671855630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5c7002ff10 msgr2=0x7f5c700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:12.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.571+0000 7f5c8759e700 1 --2- 172.21.15.173:0/1671855630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5c7002ff10 0x7f5c700323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f5c7c0096a0 tx=0x7f5c7c006b40).stop 2024-02-19T21:55:12.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.571+0000 7f5c8759e700 1 -- 172.21.15.173:0/1671855630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c880f51a0 msgr2=0x7f5c881087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:12.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.571+0000 7f5c8759e700 1 --2- 172.21.15.173:0/1671855630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c880f51a0 0x7f5c881087a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f5c780089f0 tx=0x7f5c7800b040).stop 2024-02-19T21:55:12.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.572+0000 7f5c8759e700 1 -- 172.21.15.173:0/1671855630 shutdown_connections 2024-02-19T21:55:12.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.572+0000 7f5c8759e700 1 --2- 172.21.15.173:0/1671855630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5c7002ff10 0x7f5c700323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:12.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.572+0000 7f5c8759e700 1 --2- 172.21.15.173:0/1671855630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c880f51a0 0x7f5c881087a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:12.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.572+0000 7f5c8759e700 1 -- 172.21.15.173:0/1671855630 >> 172.21.15.173:0/1671855630 conn(0x7f5c880f09a0 msgr2=0x7f5c880f2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:12.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.572+0000 7f5c8759e700 1 -- 172.21.15.173:0/1671855630 shutdown_connections 2024-02-19T21:55:12.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:12.572+0000 7f5c8759e700 1 -- 172.21.15.173:0/1671855630 wait complete. 2024-02-19T21:55:12.573 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:13.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:13 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:12.570963+0000 mon.smithi143 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.173:0/1671855630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:12.447245+0000 mgr.smithi143.bissbc (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:14.447549+0000 mgr.smithi143.bissbc (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:18.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:18.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:16.447823+0000 mgr.smithi143.bissbc (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:19.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:19 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:18.448067+0000 mgr.smithi143.bissbc (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:20.448299+0000 mgr.smithi143.bissbc (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:20.560133+0000 mon.smithi143 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:55:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:20.561287+0000 mon.smithi143 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T21:55:23.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:22.448611+0000 mgr.smithi143.bissbc (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:24.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:24.448913+0000 mgr.smithi143.bissbc (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:25.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.417+0000 7f84abd40700 1 -- 172.21.15.173:0/2340557163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a40f5410 msgr2=0x7f84a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:25.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.417+0000 7f84abd40700 1 --2- 172.21.15.173:0/2340557163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a40f5410 0x7f84a40f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f8494009fb0 tx=0x7f8494009b30).stop 2024-02-19T21:55:25.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.417+0000 7f84abd40700 1 -- 172.21.15.173:0/2340557163 shutdown_connections 2024-02-19T21:55:25.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.417+0000 7f84abd40700 1 --2- 172.21.15.173:0/2340557163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a40f5410 0x7f84a40f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:25.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.417+0000 7f84abd40700 1 -- 172.21.15.173:0/2340557163 >> 172.21.15.173:0/2340557163 conn(0x7f84a40f0b70 msgr2=0x7f84a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:25.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.418+0000 7f84abd40700 1 -- 172.21.15.173:0/2340557163 shutdown_connections 2024-02-19T21:55:25.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.418+0000 7f84abd40700 1 -- 172.21.15.173:0/2340557163 wait complete. 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.418+0000 7f84abd40700 1 Processor -- start 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.418+0000 7f84abd40700 1 -- start start 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.418+0000 7f84abd40700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a40f5410 0x7f84a40fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.418+0000 7f84abd40700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a40fb960 con 0x7f84a40f5410 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.418+0000 7f84a9adc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a40f5410 0x7f84a40fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.418+0000 7f84a9adc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a40f5410 0x7f84a40fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:44708/0 (socket says 172.21.15.173:44708) 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.418+0000 7f84a9adc700 1 -- 172.21.15.173:0/15899095 learned_addr learned my addr 172.21.15.173:0/15899095 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.419+0000 7f84a9adc700 1 -- 172.21.15.173:0/15899095 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8494005040 con 0x7f84a40f5410 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.419+0000 7f84a9adc700 1 --2- 172.21.15.173:0/15899095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a40f5410 0x7f84a40fd4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f84940089f0 tx=0x7f849400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.419+0000 7f849a7fc700 1 -- 172.21.15.173:0/15899095 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8494009320 con 0x7f84a40f5410 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.419+0000 7f849a7fc700 1 -- 172.21.15.173:0/15899095 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8494013380 con 0x7f84a40f5410 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.419+0000 7f84abd40700 1 -- 172.21.15.173:0/15899095 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a40fbb00 con 0x7f84a40f5410 2024-02-19T21:55:25.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.419+0000 7f849a7fc700 1 -- 172.21.15.173:0/15899095 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f849401ed80 con 0x7f84a40f5410 2024-02-19T21:55:25.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.419+0000 7f84abd40700 1 -- 172.21.15.173:0/15899095 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a40fbf90 con 0x7f84a40f5410 2024-02-19T21:55:25.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.420+0000 7f849a7fc700 1 -- 172.21.15.173:0/15899095 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f849401e420 con 0x7f84a40f5410 2024-02-19T21:55:25.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.420+0000 7f849a7fc700 1 --2- 172.21.15.173:0/15899095 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f849002ff60 0x7f8490032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:25.422 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.420+0000 7f849a7fc700 1 -- 172.21.15.173:0/15899095 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8494041030 con 0x7f84a40f5410 2024-02-19T21:55:25.422 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.420+0000 7f84abd40700 1 -- 172.21.15.173:0/15899095 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a4059100 con 0x7f84a40f5410 2024-02-19T21:55:25.422 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.420+0000 7f84a92db700 1 --2- 172.21.15.173:0/15899095 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f849002ff60 0x7f8490032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:25.422 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.421+0000 7f84a92db700 1 --2- 172.21.15.173:0/15899095 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f849002ff60 0x7f8490032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f84a00096a0 tx=0x7f84a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:25.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.425+0000 7f849a7fc700 1 -- 172.21.15.173:0/15899095 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f849401cb10 con 0x7f84a40f5410 2024-02-19T21:55:25.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.631+0000 7f84abd40700 1 -- 172.21.15.173:0/15899095 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84a4045ad0 con 0x7f84a40f5410 2024-02-19T21:55:25.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.631+0000 7f849a7fc700 1 -- 172.21.15.173:0/15899095 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8494004490 con 0x7f84a40f5410 2024-02-19T21:55:25.632 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:25.632 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:25.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.633+0000 7f84abd40700 1 -- 172.21.15.173:0/15899095 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f849002ff60 msgr2=0x7f8490032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:25.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.633+0000 7f84abd40700 1 --2- 172.21.15.173:0/15899095 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f849002ff60 0x7f8490032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f84a00096a0 tx=0x7f84a0006b40).stop 2024-02-19T21:55:25.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.633+0000 7f84abd40700 1 -- 172.21.15.173:0/15899095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a40f5410 msgr2=0x7f84a40fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:25.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.633+0000 7f84abd40700 1 --2- 172.21.15.173:0/15899095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a40f5410 0x7f84a40fd4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f84940089f0 tx=0x7f849400b040).stop 2024-02-19T21:55:25.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.633+0000 7f84abd40700 1 -- 172.21.15.173:0/15899095 shutdown_connections 2024-02-19T21:55:25.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.633+0000 7f84abd40700 1 --2- 172.21.15.173:0/15899095 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f849002ff60 0x7f8490032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:25.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.633+0000 7f84abd40700 1 --2- 172.21.15.173:0/15899095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84a40f5410 0x7f84a40fd4e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:25.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.633+0000 7f84abd40700 1 -- 172.21.15.173:0/15899095 >> 172.21.15.173:0/15899095 conn(0x7f84a40f0b70 msgr2=0x7f84a40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:25.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.633+0000 7f84abd40700 1 -- 172.21.15.173:0/15899095 shutdown_connections 2024-02-19T21:55:25.633 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:25.633+0000 7f84abd40700 1 -- 172.21.15.173:0/15899095 wait complete. 2024-02-19T21:55:25.634 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:26.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:25 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:25.632772+0000 mon.smithi143 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.173:0/15899095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:27.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:27.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:26.449197+0000 mgr.smithi143.bissbc (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:29.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.235+0000 7f44aeb34700 1 -- 172.21.15.173:0/2495042649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44a8060280 msgr2=0x7f44a80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:29.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.235+0000 7f44aeb34700 1 --2- 172.21.15.173:0/2495042649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44a8060280 0x7f44a80f90f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f449c009fb0 tx=0x7f449c009b30).stop 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.235+0000 7f44aeb34700 1 -- 172.21.15.173:0/2495042649 shutdown_connections 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.235+0000 7f44aeb34700 1 --2- 172.21.15.173:0/2495042649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44a8060280 0x7f44a80f90f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.235+0000 7f44aeb34700 1 -- 172.21.15.173:0/2495042649 >> 172.21.15.173:0/2495042649 conn(0x7f44a80f0b70 msgr2=0x7f44a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.235+0000 7f44aeb34700 1 -- 172.21.15.173:0/2495042649 shutdown_connections 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.235+0000 7f44aeb34700 1 -- 172.21.15.173:0/2495042649 wait complete. 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.236+0000 7f44aeb34700 1 Processor -- start 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.236+0000 7f44aeb34700 1 -- start start 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.236+0000 7f44aeb34700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44a8060280 0x7f44a8106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.236+0000 7f44aeb34700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8106c70 con 0x7f44a8060280 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.236+0000 7f44ac8d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44a8060280 0x7f44a8106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.236+0000 7f44ac8d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44a8060280 0x7f44a8106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:44718/0 (socket says 172.21.15.173:44718) 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.236+0000 7f44ac8d0700 1 -- 172.21.15.173:0/2478974092 learned_addr learned my addr 172.21.15.173:0/2478974092 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:29.238 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.237+0000 7f44ac8d0700 1 -- 172.21.15.173:0/2478974092 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c005040 con 0x7f44a8060280 2024-02-19T21:55:29.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.237+0000 7f44ac8d0700 1 --2- 172.21.15.173:0/2478974092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44a8060280 0x7f44a8106780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f449c0089f0 tx=0x7f449c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:29.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.237+0000 7f44a57fa700 1 -- 172.21.15.173:0/2478974092 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f449c009320 con 0x7f44a8060280 2024-02-19T21:55:29.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.237+0000 7f44a57fa700 1 -- 172.21.15.173:0/2478974092 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f449c013380 con 0x7f44a8060280 2024-02-19T21:55:29.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.237+0000 7f44aeb34700 1 -- 172.21.15.173:0/2478974092 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a8106e70 con 0x7f44a8060280 2024-02-19T21:55:29.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.237+0000 7f44a57fa700 1 -- 172.21.15.173:0/2478974092 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f449c014e20 con 0x7f44a8060280 2024-02-19T21:55:29.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.237+0000 7f44aeb34700 1 -- 172.21.15.173:0/2478974092 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a8107300 con 0x7f44a8060280 2024-02-19T21:55:29.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.238+0000 7f44a57fa700 1 -- 172.21.15.173:0/2478974092 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f449c014420 con 0x7f44a8060280 2024-02-19T21:55:29.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.238+0000 7f44a57fa700 1 --2- 172.21.15.173:0/2478974092 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f449802ff10 0x7f44980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:29.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.238+0000 7f44aeb34700 1 -- 172.21.15.173:0/2478974092 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a8045ad0 con 0x7f44a8060280 2024-02-19T21:55:29.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.238+0000 7f44a57fa700 1 -- 172.21.15.173:0/2478974092 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f449c040f50 con 0x7f44a8060280 2024-02-19T21:55:29.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.238+0000 7f44a7fff700 1 --2- 172.21.15.173:0/2478974092 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f449802ff10 0x7f44980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:29.240 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.239+0000 7f44a7fff700 1 --2- 172.21.15.173:0/2478974092 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f449802ff10 0x7f44980323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f44940096a0 tx=0x7f4494006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:29.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.243+0000 7f44a57fa700 1 -- 172.21.15.173:0/2478974092 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f449c0095f0 con 0x7f44a8060280 2024-02-19T21:55:29.449 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.448+0000 7f44aeb34700 1 -- 172.21.15.173:0/2478974092 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44a81081b0 con 0x7f44a8060280 2024-02-19T21:55:29.449 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.450+0000 7f44a57fa700 1 -- 172.21.15.173:0/2478974092 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f449c01db70 con 0x7f44a8060280 2024-02-19T21:55:29.450 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:29.450 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.450+0000 7f44aeb34700 1 -- 172.21.15.173:0/2478974092 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f449802ff10 msgr2=0x7f44980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.450+0000 7f44aeb34700 1 --2- 172.21.15.173:0/2478974092 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f449802ff10 0x7f44980323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f44940096a0 tx=0x7f4494006b40).stop 2024-02-19T21:55:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.450+0000 7f44aeb34700 1 -- 172.21.15.173:0/2478974092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44a8060280 msgr2=0x7f44a8106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.450+0000 7f44aeb34700 1 --2- 172.21.15.173:0/2478974092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44a8060280 0x7f44a8106780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f449c0089f0 tx=0x7f449c00b040).stop 2024-02-19T21:55:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.450+0000 7f44aeb34700 1 -- 172.21.15.173:0/2478974092 shutdown_connections 2024-02-19T21:55:29.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.450+0000 7f44aeb34700 1 --2- 172.21.15.173:0/2478974092 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f449802ff10 0x7f44980323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:29.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.450+0000 7f44aeb34700 1 --2- 172.21.15.173:0/2478974092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44a8060280 0x7f44a8106780 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:29.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.450+0000 7f44aeb34700 1 -- 172.21.15.173:0/2478974092 >> 172.21.15.173:0/2478974092 conn(0x7f44a80f0b70 msgr2=0x7f44a80f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:29.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.450+0000 7f44aeb34700 1 -- 172.21.15.173:0/2478974092 shutdown_connections 2024-02-19T21:55:29.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:29.451+0000 7f44aeb34700 1 -- 172.21.15.173:0/2478974092 wait complete. 2024-02-19T21:55:29.452 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:29.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:28.449530+0000 mgr.smithi143.bissbc (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:30.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:29.450825+0000 mon.smithi143 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.173:0/2478974092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:31.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:31.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:32.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:30.449930+0000 mgr.smithi143.bissbc (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:33.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.490+0000 7f1514e32700 1 -- 172.21.15.173:0/2328845156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15100f5410 msgr2=0x7f15100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.490+0000 7f1514e32700 1 --2- 172.21.15.173:0/2328845156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15100f5410 0x7f15100f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f14f8009fb0 tx=0x7f14f8009b30).stop 2024-02-19T21:55:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.490+0000 7f1514e32700 1 -- 172.21.15.173:0/2328845156 shutdown_connections 2024-02-19T21:55:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.490+0000 7f1514e32700 1 --2- 172.21.15.173:0/2328845156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15100f5410 0x7f15100f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.490+0000 7f1514e32700 1 -- 172.21.15.173:0/2328845156 >> 172.21.15.173:0/2328845156 conn(0x7f15100f0b70 msgr2=0x7f15100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.490+0000 7f1514e32700 1 -- 172.21.15.173:0/2328845156 shutdown_connections 2024-02-19T21:55:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.490+0000 7f1514e32700 1 -- 172.21.15.173:0/2328845156 wait complete. 2024-02-19T21:55:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.491+0000 7f1514e32700 1 Processor -- start 2024-02-19T21:55:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.491+0000 7f1514e32700 1 -- start start 2024-02-19T21:55:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.491+0000 7f1514e32700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15100f5410 0x7f1510109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.491+0000 7f1514e32700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1510060220 con 0x7f15100f5410 2024-02-19T21:55:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.491+0000 7f150e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15100f5410 0x7f1510109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.491+0000 7f150e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15100f5410 0x7f1510109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:60598/0 (socket says 172.21.15.173:60598) 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.491+0000 7f150e59c700 1 -- 172.21.15.173:0/2523092395 learned_addr learned my addr 172.21.15.173:0/2523092395 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.491+0000 7f150e59c700 1 -- 172.21.15.173:0/2523092395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f8005040 con 0x7f15100f5410 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.491+0000 7f150e59c700 1 --2- 172.21.15.173:0/2523092395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15100f5410 0x7f1510109500 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f14f8005010 tx=0x7f14f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.492+0000 7f1506ffd700 1 -- 172.21.15.173:0/2523092395 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14f8009320 con 0x7f15100f5410 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.492+0000 7f1506ffd700 1 -- 172.21.15.173:0/2523092395 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14f8013380 con 0x7f15100f5410 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.492+0000 7f1514e32700 1 -- 172.21.15.173:0/2523092395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1510060420 con 0x7f15100f5410 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.492+0000 7f1506ffd700 1 -- 172.21.15.173:0/2523092395 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14f8014e50 con 0x7f15100f5410 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.492+0000 7f1514e32700 1 -- 172.21.15.173:0/2523092395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1510109bb0 con 0x7f15100f5410 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.492+0000 7f1506ffd700 1 -- 172.21.15.173:0/2523092395 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14f8014420 con 0x7f15100f5410 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.492+0000 7f1514e32700 1 -- 172.21.15.173:0/2523092395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1510059100 con 0x7f15100f5410 2024-02-19T21:55:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.493+0000 7f1506ffd700 1 --2- 172.21.15.173:0/2523092395 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14fc02ff10 0x7f14fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:33.495 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.493+0000 7f1506ffd700 1 -- 172.21.15.173:0/2523092395 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14f8040eb0 con 0x7f15100f5410 2024-02-19T21:55:33.495 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.493+0000 7f150dd9b700 1 --2- 172.21.15.173:0/2523092395 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14fc02ff10 0x7f14fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:33.495 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.493+0000 7f150dd9b700 1 --2- 172.21.15.173:0/2523092395 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14fc02ff10 0x7f14fc0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f15000096a0 tx=0x7f1500006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:33.497 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.498+0000 7f1506ffd700 1 -- 172.21.15.173:0/2523092395 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14f80095f0 con 0x7f15100f5410 2024-02-19T21:55:33.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.704+0000 7f1514e32700 1 -- 172.21.15.173:0/2523092395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1510045ad0 con 0x7f15100f5410 2024-02-19T21:55:33.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.705+0000 7f1506ffd700 1 -- 172.21.15.173:0/2523092395 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f14f801db70 con 0x7f15100f5410 2024-02-19T21:55:33.704 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:33.705 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:33.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.706+0000 7f1514e32700 1 -- 172.21.15.173:0/2523092395 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14fc02ff10 msgr2=0x7f14fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:33.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.706+0000 7f1514e32700 1 --2- 172.21.15.173:0/2523092395 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14fc02ff10 0x7f14fc0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f15000096a0 tx=0x7f1500006b40).stop 2024-02-19T21:55:33.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.706+0000 7f1514e32700 1 -- 172.21.15.173:0/2523092395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15100f5410 msgr2=0x7f1510109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:33.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.706+0000 7f1514e32700 1 --2- 172.21.15.173:0/2523092395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15100f5410 0x7f1510109500 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f14f8005010 tx=0x7f14f800b040).stop 2024-02-19T21:55:33.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.706+0000 7f1514e32700 1 -- 172.21.15.173:0/2523092395 shutdown_connections 2024-02-19T21:55:33.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.706+0000 7f1514e32700 1 --2- 172.21.15.173:0/2523092395 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14fc02ff10 0x7f14fc0323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:33.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.706+0000 7f1514e32700 1 --2- 172.21.15.173:0/2523092395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15100f5410 0x7f1510109500 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:33.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.706+0000 7f1514e32700 1 -- 172.21.15.173:0/2523092395 >> 172.21.15.173:0/2523092395 conn(0x7f15100f0b70 msgr2=0x7f15100f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:33.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.706+0000 7f1514e32700 1 -- 172.21.15.173:0/2523092395 shutdown_connections 2024-02-19T21:55:33.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:33.706+0000 7f1514e32700 1 -- 172.21.15.173:0/2523092395 wait complete. 2024-02-19T21:55:33.707 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:32.450213+0000 mgr.smithi143.bissbc (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:33.705658+0000 mon.smithi143 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.173:0/2523092395' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:35.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:35.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:34.450552+0000 mgr.smithi143.bissbc (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:37.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.203+0000 7fc451da8700 1 -- 172.21.15.173:0/2774487288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c0f6510 msgr2=0x7fc44c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:37.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.203+0000 7fc451da8700 1 --2- 172.21.15.173:0/2774487288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c0f6510 0x7fc44c0f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fc440009fb0 tx=0x7fc440009b30).stop 2024-02-19T21:55:37.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.203+0000 7fc451da8700 1 -- 172.21.15.173:0/2774487288 shutdown_connections 2024-02-19T21:55:37.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.203+0000 7fc451da8700 1 --2- 172.21.15.173:0/2774487288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c0f6510 0x7fc44c0f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:37.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.203+0000 7fc451da8700 1 -- 172.21.15.173:0/2774487288 >> 172.21.15.173:0/2774487288 conn(0x7fc44c0f1c70 msgr2=0x7fc44c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:37.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.203+0000 7fc451da8700 1 -- 172.21.15.173:0/2774487288 shutdown_connections 2024-02-19T21:55:37.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.203+0000 7fc451da8700 1 -- 172.21.15.173:0/2774487288 wait complete. 2024-02-19T21:55:37.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.203+0000 7fc451da8700 1 Processor -- start 2024-02-19T21:55:37.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.204+0000 7fc451da8700 1 -- start start 2024-02-19T21:55:37.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.204+0000 7fc451da8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c0f6510 0x7fc44c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:37.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.204+0000 7fc451da8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c069e20 con 0x7fc44c0f6510 2024-02-19T21:55:37.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.204+0000 7fc44b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c0f6510 0x7fc44c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:37.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.204+0000 7fc44b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c0f6510 0x7fc44c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:60620/0 (socket says 172.21.15.173:60620) 2024-02-19T21:55:37.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.204+0000 7fc44b7fe700 1 -- 172.21.15.173:0/231928418 learned_addr learned my addr 172.21.15.173:0/231928418 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:37.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.204+0000 7fc44b7fe700 1 -- 172.21.15.173:0/231928418 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc440005040 con 0x7fc44c0f6510 2024-02-19T21:55:37.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.204+0000 7fc44b7fe700 1 --2- 172.21.15.173:0/231928418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c0f6510 0x7fc44c06b9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc440005010 tx=0x7fc44000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:37.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.205+0000 7fc433fff700 1 -- 172.21.15.173:0/231928418 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc440009320 con 0x7fc44c0f6510 2024-02-19T21:55:37.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.205+0000 7fc433fff700 1 -- 172.21.15.173:0/231928418 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc440013380 con 0x7fc44c0f6510 2024-02-19T21:55:37.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.205+0000 7fc451da8700 1 -- 172.21.15.173:0/231928418 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c069fc0 con 0x7fc44c0f6510 2024-02-19T21:55:37.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.205+0000 7fc433fff700 1 -- 172.21.15.173:0/231928418 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc440014e50 con 0x7fc44c0f6510 2024-02-19T21:55:37.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.205+0000 7fc451da8700 1 -- 172.21.15.173:0/231928418 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44c06a450 con 0x7fc44c0f6510 2024-02-19T21:55:37.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.206+0000 7fc433fff700 1 -- 172.21.15.173:0/231928418 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4400095f0 con 0x7fc44c0f6510 2024-02-19T21:55:37.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.206+0000 7fc451da8700 1 -- 172.21.15.173:0/231928418 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44c059100 con 0x7fc44c0f6510 2024-02-19T21:55:37.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.206+0000 7fc433fff700 1 --2- 172.21.15.173:0/231928418 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc43402ff10 0x7fc4340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:37.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.206+0000 7fc433fff700 1 -- 172.21.15.173:0/231928418 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc440040f00 con 0x7fc44c0f6510 2024-02-19T21:55:37.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.206+0000 7fc44affd700 1 --2- 172.21.15.173:0/231928418 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc43402ff10 0x7fc4340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:37.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.206+0000 7fc44affd700 1 --2- 172.21.15.173:0/231928418 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc43402ff10 0x7fc4340323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc43c0096a0 tx=0x7fc43c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:37.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.211+0000 7fc433fff700 1 -- 172.21.15.173:0/231928418 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc440014420 con 0x7fc44c0f6510 2024-02-19T21:55:37.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.414+0000 7fc451da8700 1 -- 172.21.15.173:0/231928418 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc44c0fc0a0 con 0x7fc44c0f6510 2024-02-19T21:55:37.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.414+0000 7fc433fff700 1 -- 172.21.15.173:0/231928418 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc44001d7f0 con 0x7fc44c0f6510 2024-02-19T21:55:37.415 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:37.415 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:37.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.416+0000 7fc451da8700 1 -- 172.21.15.173:0/231928418 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc43402ff10 msgr2=0x7fc4340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:37.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.416+0000 7fc451da8700 1 --2- 172.21.15.173:0/231928418 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc43402ff10 0x7fc4340323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc43c0096a0 tx=0x7fc43c006b40).stop 2024-02-19T21:55:37.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.416+0000 7fc451da8700 1 -- 172.21.15.173:0/231928418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c0f6510 msgr2=0x7fc44c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:37.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.416+0000 7fc451da8700 1 --2- 172.21.15.173:0/231928418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c0f6510 0x7fc44c06b9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc440005010 tx=0x7fc44000b040).stop 2024-02-19T21:55:37.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.416+0000 7fc451da8700 1 -- 172.21.15.173:0/231928418 shutdown_connections 2024-02-19T21:55:37.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.416+0000 7fc451da8700 1 --2- 172.21.15.173:0/231928418 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc43402ff10 0x7fc4340323d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:37.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.416+0000 7fc451da8700 1 --2- 172.21.15.173:0/231928418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc44c0f6510 0x7fc44c06b9a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:37.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.416+0000 7fc451da8700 1 -- 172.21.15.173:0/231928418 >> 172.21.15.173:0/231928418 conn(0x7fc44c0f1c70 msgr2=0x7fc44c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:37.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.416+0000 7fc451da8700 1 -- 172.21.15.173:0/231928418 shutdown_connections 2024-02-19T21:55:37.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:37.416+0000 7fc451da8700 1 -- 172.21.15.173:0/231928418 wait complete. 2024-02-19T21:55:37.417 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:37.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:37 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:37.415654+0000 mon.smithi143 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.173:0/231928418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:36.450927+0000 mgr.smithi143.bissbc (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:39.083 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:39.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:38.451210+0000 mgr.smithi143.bissbc (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:40.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.956+0000 7f8605e0c700 1 -- 172.21.15.173:0/2381660760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8600060280 msgr2=0x7f86000f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:40.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.956+0000 7f8605e0c700 1 --2- 172.21.15.173:0/2381660760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8600060280 0x7f86000f90f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f85e8009fb0 tx=0x7f85e8009b30).stop 2024-02-19T21:55:40.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.956+0000 7f8605e0c700 1 -- 172.21.15.173:0/2381660760 shutdown_connections 2024-02-19T21:55:40.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.956+0000 7f8605e0c700 1 --2- 172.21.15.173:0/2381660760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8600060280 0x7f86000f90f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:40.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.956+0000 7f8605e0c700 1 -- 172.21.15.173:0/2381660760 >> 172.21.15.173:0/2381660760 conn(0x7f86000f0b70 msgr2=0x7f86000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:40.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.956+0000 7f8605e0c700 1 -- 172.21.15.173:0/2381660760 shutdown_connections 2024-02-19T21:55:40.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.956+0000 7f8605e0c700 1 -- 172.21.15.173:0/2381660760 wait complete. 2024-02-19T21:55:40.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.957+0000 7f8605e0c700 1 Processor -- start 2024-02-19T21:55:40.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.957+0000 7f8605e0c700 1 -- start start 2024-02-19T21:55:40.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.957+0000 7f8605e0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8600060280 0x7f8600106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:40.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.957+0000 7f8605e0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8600106c70 con 0x7f8600060280 2024-02-19T21:55:40.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.957+0000 7f85ff7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8600060280 0x7f8600106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:40.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.957+0000 7f85ff7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8600060280 0x7f8600106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:60634/0 (socket says 172.21.15.173:60634) 2024-02-19T21:55:40.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.957+0000 7f85ff7fe700 1 -- 172.21.15.173:0/1304401244 learned_addr learned my addr 172.21.15.173:0/1304401244 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:40.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.958+0000 7f85ff7fe700 1 -- 172.21.15.173:0/1304401244 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e8005040 con 0x7f8600060280 2024-02-19T21:55:40.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.958+0000 7f85ff7fe700 1 --2- 172.21.15.173:0/1304401244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8600060280 0x7f8600106780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f85e80089f0 tx=0x7f85e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:40.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.958+0000 7f85f7fff700 1 -- 172.21.15.173:0/1304401244 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85e8009320 con 0x7f8600060280 2024-02-19T21:55:40.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.958+0000 7f8605e0c700 1 -- 172.21.15.173:0/1304401244 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8600106e70 con 0x7f8600060280 2024-02-19T21:55:40.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.958+0000 7f85f7fff700 1 -- 172.21.15.173:0/1304401244 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85e8013380 con 0x7f8600060280 2024-02-19T21:55:40.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.958+0000 7f85f7fff700 1 -- 172.21.15.173:0/1304401244 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85e8014e20 con 0x7f8600060280 2024-02-19T21:55:40.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.958+0000 7f8605e0c700 1 -- 172.21.15.173:0/1304401244 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8600107300 con 0x7f8600060280 2024-02-19T21:55:40.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.959+0000 7f85f7fff700 1 -- 172.21.15.173:0/1304401244 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85e80263a0 con 0x7f8600060280 2024-02-19T21:55:40.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.959+0000 7f8605e0c700 1 -- 172.21.15.173:0/1304401244 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8600045ad0 con 0x7f8600060280 2024-02-19T21:55:40.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.959+0000 7f85f7fff700 1 --2- 172.21.15.173:0/1304401244 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f85ec02ff10 0x7f85ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:40.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.959+0000 7f85f7fff700 1 -- 172.21.15.173:0/1304401244 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85e80411e0 con 0x7f8600060280 2024-02-19T21:55:40.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.959+0000 7f85feffd700 1 --2- 172.21.15.173:0/1304401244 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f85ec02ff10 0x7f85ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:40.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.959+0000 7f85feffd700 1 --2- 172.21.15.173:0/1304401244 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f85ec02ff10 0x7f85ec0323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f85f00096a0 tx=0x7f85f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:40.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:40.964+0000 7f85f7fff700 1 -- 172.21.15.173:0/1304401244 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85e801d470 con 0x7f8600060280 2024-02-19T21:55:41.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.168+0000 7f8605e0c700 1 -- 172.21.15.173:0/1304401244 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86001081b0 con 0x7f8600060280 2024-02-19T21:55:41.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.169+0000 7f85f7fff700 1 -- 172.21.15.173:0/1304401244 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f85e80126f0 con 0x7f8600060280 2024-02-19T21:55:41.169 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:41.169 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:41.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.170+0000 7f8605e0c700 1 -- 172.21.15.173:0/1304401244 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f85ec02ff10 msgr2=0x7f85ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:41.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.170+0000 7f8605e0c700 1 --2- 172.21.15.173:0/1304401244 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f85ec02ff10 0x7f85ec0323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f85f00096a0 tx=0x7f85f0006b40).stop 2024-02-19T21:55:41.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.170+0000 7f8605e0c700 1 -- 172.21.15.173:0/1304401244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8600060280 msgr2=0x7f8600106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:41.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.170+0000 7f8605e0c700 1 --2- 172.21.15.173:0/1304401244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8600060280 0x7f8600106780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f85e80089f0 tx=0x7f85e800b040).stop 2024-02-19T21:55:41.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.170+0000 7f8605e0c700 1 -- 172.21.15.173:0/1304401244 shutdown_connections 2024-02-19T21:55:41.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.170+0000 7f8605e0c700 1 --2- 172.21.15.173:0/1304401244 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f85ec02ff10 0x7f85ec0323d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:41.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.170+0000 7f8605e0c700 1 --2- 172.21.15.173:0/1304401244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8600060280 0x7f8600106780 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:41.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.170+0000 7f8605e0c700 1 -- 172.21.15.173:0/1304401244 >> 172.21.15.173:0/1304401244 conn(0x7f86000f0b70 msgr2=0x7f86000f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:41.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.170+0000 7f8605e0c700 1 -- 172.21.15.173:0/1304401244 shutdown_connections 2024-02-19T21:55:41.171 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:41.170+0000 7f8605e0c700 1 -- 172.21.15.173:0/1304401244 wait complete. 2024-02-19T21:55:41.171 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:41.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:41 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:41.170016+0000 mon.smithi143 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.173:0/1304401244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:42.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:42.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:40.451553+0000 mgr.smithi143.bissbc (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:44.574 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.572+0000 7ff5a261b700 1 -- 172.21.15.173:0/22315041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff59c0f51a0 msgr2=0x7ff59c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:44.574 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.572+0000 7ff5a261b700 1 --2- 172.21.15.173:0/22315041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff59c0f51a0 0x7ff59c0f55c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7ff58c009fb0 tx=0x7ff58c009b30).stop 2024-02-19T21:55:44.574 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.572+0000 7ff5a261b700 1 -- 172.21.15.173:0/22315041 shutdown_connections 2024-02-19T21:55:44.574 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.572+0000 7ff5a261b700 1 --2- 172.21.15.173:0/22315041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff59c0f51a0 0x7ff59c0f55c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:44.574 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.572+0000 7ff5a261b700 1 -- 172.21.15.173:0/22315041 >> 172.21.15.173:0/22315041 conn(0x7ff59c0f09a0 msgr2=0x7ff59c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.573+0000 7ff5a261b700 1 -- 172.21.15.173:0/22315041 shutdown_connections 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.573+0000 7ff5a261b700 1 -- 172.21.15.173:0/22315041 wait complete. 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.573+0000 7ff5a261b700 1 Processor -- start 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.573+0000 7ff5a261b700 1 -- start start 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.573+0000 7ff5a261b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff59c0f51a0 0x7ff59c109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.573+0000 7ff5a261b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c060280 con 0x7ff59c0f51a0 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.573+0000 7ff5a1619700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff59c0f51a0 0x7ff59c109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.574+0000 7ff5a1619700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff59c0f51a0 0x7ff59c109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:35392/0 (socket says 172.21.15.173:35392) 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.574+0000 7ff5a1619700 1 -- 172.21.15.173:0/4126886303 learned_addr learned my addr 172.21.15.173:0/4126886303 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.574+0000 7ff5a1619700 1 -- 172.21.15.173:0/4126886303 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c005040 con 0x7ff59c0f51a0 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.574+0000 7ff5a1619700 1 --2- 172.21.15.173:0/4126886303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff59c0f51a0 0x7ff59c109290 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff58c009830 tx=0x7ff58c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.574+0000 7ff591ffb700 1 -- 172.21.15.173:0/4126886303 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff58c009320 con 0x7ff59c0f51a0 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.574+0000 7ff591ffb700 1 -- 172.21.15.173:0/4126886303 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff58c013380 con 0x7ff59c0f51a0 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.574+0000 7ff5a261b700 1 -- 172.21.15.173:0/4126886303 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c060480 con 0x7ff59c0f51a0 2024-02-19T21:55:44.575 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.574+0000 7ff591ffb700 1 -- 172.21.15.173:0/4126886303 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff58c014420 con 0x7ff59c0f51a0 2024-02-19T21:55:44.576 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.574+0000 7ff5a261b700 1 -- 172.21.15.173:0/4126886303 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c1099c0 con 0x7ff59c0f51a0 2024-02-19T21:55:44.576 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.575+0000 7ff591ffb700 1 -- 172.21.15.173:0/4126886303 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff58c01e420 con 0x7ff59c0f51a0 2024-02-19T21:55:44.577 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.575+0000 7ff5a261b700 1 -- 172.21.15.173:0/4126886303 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59c045ad0 con 0x7ff59c0f51a0 2024-02-19T21:55:44.577 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.575+0000 7ff591ffb700 1 --2- 172.21.15.173:0/4126886303 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff58802ff60 0x7ff588032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:44.577 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.575+0000 7ff591ffb700 1 -- 172.21.15.173:0/4126886303 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff58c040ef0 con 0x7ff59c0f51a0 2024-02-19T21:55:44.577 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.575+0000 7ff5a0e18700 1 --2- 172.21.15.173:0/4126886303 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff58802ff60 0x7ff588032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:44.577 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.576+0000 7ff5a0e18700 1 --2- 172.21.15.173:0/4126886303 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff58802ff60 0x7ff588032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff5980096a0 tx=0x7ff598006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:44.580 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.581+0000 7ff591ffb700 1 -- 172.21.15.173:0/4126886303 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff58c01cb10 con 0x7ff59c0f51a0 2024-02-19T21:55:44.783 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.783+0000 7ff5a261b700 1 -- 172.21.15.173:0/4126886303 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff59c0fad30 con 0x7ff59c0f51a0 2024-02-19T21:55:44.783 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.784+0000 7ff591ffb700 1 -- 172.21.15.173:0/4126886303 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff58c02c2d0 con 0x7ff59c0f51a0 2024-02-19T21:55:44.784 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:44.784 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:44.785 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.785+0000 7ff5a261b700 1 -- 172.21.15.173:0/4126886303 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff58802ff60 msgr2=0x7ff588032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:44.785 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.785+0000 7ff5a261b700 1 --2- 172.21.15.173:0/4126886303 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff58802ff60 0x7ff588032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff5980096a0 tx=0x7ff598006b40).stop 2024-02-19T21:55:44.785 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.785+0000 7ff5a261b700 1 -- 172.21.15.173:0/4126886303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff59c0f51a0 msgr2=0x7ff59c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:44.785 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.785+0000 7ff5a261b700 1 --2- 172.21.15.173:0/4126886303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff59c0f51a0 0x7ff59c109290 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff58c009830 tx=0x7ff58c00b040).stop 2024-02-19T21:55:44.786 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.785+0000 7ff5a261b700 1 -- 172.21.15.173:0/4126886303 shutdown_connections 2024-02-19T21:55:44.786 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.785+0000 7ff5a261b700 1 --2- 172.21.15.173:0/4126886303 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff58802ff60 0x7ff588032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:44.786 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.785+0000 7ff5a261b700 1 --2- 172.21.15.173:0/4126886303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff59c0f51a0 0x7ff59c109290 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:44.786 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.785+0000 7ff5a261b700 1 -- 172.21.15.173:0/4126886303 >> 172.21.15.173:0/4126886303 conn(0x7ff59c0f09a0 msgr2=0x7ff59c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:44.786 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.785+0000 7ff5a261b700 1 -- 172.21.15.173:0/4126886303 shutdown_connections 2024-02-19T21:55:44.786 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:44.785+0000 7ff5a261b700 1 -- 172.21.15.173:0/4126886303 wait complete. 2024-02-19T21:55:44.786 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:42.451817+0000 mgr.smithi143.bissbc (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:45.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:45 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:44.452057+0000 mgr.smithi143.bissbc (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:45.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:45 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:44.784880+0000 mon.smithi143 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.173:0/4126886303' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:46.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:46.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:48.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.337+0000 7f96b30d4700 1 -- 172.21.15.173:0/22557359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96ac0f5410 msgr2=0x7f96ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:48.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.337+0000 7f96b30d4700 1 --2- 172.21.15.173:0/22557359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96ac0f5410 0x7f96ac0f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f96a0009fb0 tx=0x7f96a0009b30).stop 2024-02-19T21:55:48.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.337+0000 7f96b30d4700 1 -- 172.21.15.173:0/22557359 shutdown_connections 2024-02-19T21:55:48.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.337+0000 7f96b30d4700 1 --2- 172.21.15.173:0/22557359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96ac0f5410 0x7f96ac0f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:48.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.337+0000 7f96b30d4700 1 -- 172.21.15.173:0/22557359 >> 172.21.15.173:0/22557359 conn(0x7f96ac0f0b70 msgr2=0x7f96ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:48.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.338+0000 7f96b30d4700 1 -- 172.21.15.173:0/22557359 shutdown_connections 2024-02-19T21:55:48.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.338+0000 7f96b30d4700 1 -- 172.21.15.173:0/22557359 wait complete. 2024-02-19T21:55:48.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.338+0000 7f96b30d4700 1 Processor -- start 2024-02-19T21:55:48.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.338+0000 7f96b30d4700 1 -- start start 2024-02-19T21:55:48.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.338+0000 7f96b30d4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96ac0f5410 0x7f96ac109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:48.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.338+0000 7f96b30d4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac060220 con 0x7f96ac0f5410 2024-02-19T21:55:48.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.339+0000 7f96b0e70700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96ac0f5410 0x7f96ac109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.339+0000 7f96b0e70700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96ac0f5410 0x7f96ac109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:35418/0 (socket says 172.21.15.173:35418) 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.339+0000 7f96b0e70700 1 -- 172.21.15.173:0/2107609391 learned_addr learned my addr 172.21.15.173:0/2107609391 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.339+0000 7f96b0e70700 1 -- 172.21.15.173:0/2107609391 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a0005040 con 0x7f96ac0f5410 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.339+0000 7f96b0e70700 1 --2- 172.21.15.173:0/2107609391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96ac0f5410 0x7f96ac109500 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f96a0005010 tx=0x7f96a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.339+0000 7f96a97fa700 1 -- 172.21.15.173:0/2107609391 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96a0009320 con 0x7f96ac0f5410 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.339+0000 7f96a97fa700 1 -- 172.21.15.173:0/2107609391 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96a0013380 con 0x7f96ac0f5410 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.339+0000 7f96b30d4700 1 -- 172.21.15.173:0/2107609391 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac060420 con 0x7f96ac0f5410 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.339+0000 7f96a97fa700 1 -- 172.21.15.173:0/2107609391 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96a0014e50 con 0x7f96ac0f5410 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.339+0000 7f96b30d4700 1 -- 172.21.15.173:0/2107609391 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac109bb0 con 0x7f96ac0f5410 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.340+0000 7f96a97fa700 1 -- 172.21.15.173:0/2107609391 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96a00263a0 con 0x7f96ac0f5410 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.340+0000 7f96b30d4700 1 -- 172.21.15.173:0/2107609391 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac059100 con 0x7f96ac0f5410 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.340+0000 7f96a97fa700 1 --2- 172.21.15.173:0/2107609391 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f969c02ff10 0x7f969c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.340+0000 7f96a97fa700 1 -- 172.21.15.173:0/2107609391 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96a0041160 con 0x7f96ac0f5410 2024-02-19T21:55:48.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.341+0000 7f96abfff700 1 --2- 172.21.15.173:0/2107609391 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f969c02ff10 0x7f969c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:48.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.341+0000 7f96abfff700 1 --2- 172.21.15.173:0/2107609391 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f969c02ff10 0x7f969c0323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f96980096a0 tx=0x7f9698006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:48.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.345+0000 7f96a97fa700 1 -- 172.21.15.173:0/2107609391 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96a00126f0 con 0x7f96ac0f5410 2024-02-19T21:55:48.549 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.549+0000 7f96b30d4700 1 -- 172.21.15.173:0/2107609391 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96ac045ad0 con 0x7f96ac0f5410 2024-02-19T21:55:48.550 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.550+0000 7f96a97fa700 1 -- 172.21.15.173:0/2107609391 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f96a0012330 con 0x7f96ac0f5410 2024-02-19T21:55:48.550 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:48.550 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:48.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.552+0000 7f96b30d4700 1 -- 172.21.15.173:0/2107609391 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f969c02ff10 msgr2=0x7f969c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:48.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.552+0000 7f96b30d4700 1 --2- 172.21.15.173:0/2107609391 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f969c02ff10 0x7f969c0323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f96980096a0 tx=0x7f9698006b40).stop 2024-02-19T21:55:48.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.552+0000 7f96b30d4700 1 -- 172.21.15.173:0/2107609391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96ac0f5410 msgr2=0x7f96ac109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:48.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.552+0000 7f96b30d4700 1 --2- 172.21.15.173:0/2107609391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96ac0f5410 0x7f96ac109500 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f96a0005010 tx=0x7f96a000b040).stop 2024-02-19T21:55:48.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.552+0000 7f96b30d4700 1 -- 172.21.15.173:0/2107609391 shutdown_connections 2024-02-19T21:55:48.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.552+0000 7f96b30d4700 1 --2- 172.21.15.173:0/2107609391 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f969c02ff10 0x7f969c0323d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:48.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.552+0000 7f96b30d4700 1 --2- 172.21.15.173:0/2107609391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96ac0f5410 0x7f96ac109500 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:48.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.552+0000 7f96b30d4700 1 -- 172.21.15.173:0/2107609391 >> 172.21.15.173:0/2107609391 conn(0x7f96ac0f0b70 msgr2=0x7f96ac0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:48.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.552+0000 7f96b30d4700 1 -- 172.21.15.173:0/2107609391 shutdown_connections 2024-02-19T21:55:48.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:48.552+0000 7f96b30d4700 1 -- 172.21.15.173:0/2107609391 wait complete. 2024-02-19T21:55:48.553 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:46.452345+0000 mgr.smithi143.bissbc (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:49.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:49 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:48.551216+0000 mon.smithi143 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.173:0/2107609391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:50.253 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:50.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:48.452658+0000 mgr.smithi143.bissbc (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:51.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:51 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:50.452902+0000 mgr.smithi143.bissbc (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:52.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.081+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1452814358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d80f62a0 msgr2=0x7fe9d80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:52.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.081+0000 7fe9de7a7700 1 --2- 172.21.15.173:0/1452814358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d80f62a0 0x7fe9d80f66c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe9d4009fb0 tx=0x7fe9d4009b30).stop 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.081+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1452814358 shutdown_connections 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.081+0000 7fe9de7a7700 1 --2- 172.21.15.173:0/1452814358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d80f62a0 0x7fe9d80f66c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.081+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1452814358 >> 172.21.15.173:0/1452814358 conn(0x7fe9d80f1aa0 msgr2=0x7fe9d80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.081+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1452814358 shutdown_connections 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.081+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1452814358 wait complete. 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.081+0000 7fe9de7a7700 1 Processor -- start 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.081+0000 7fe9de7a7700 1 -- start start 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.082+0000 7fe9de7a7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d80f62a0 0x7fe9d81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.082+0000 7fe9de7a7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8106ac0 con 0x7fe9d80f62a0 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.082+0000 7fe9dd7a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d80f62a0 0x7fe9d81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.082+0000 7fe9dd7a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d80f62a0 0x7fe9d81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:59600/0 (socket says 172.21.15.173:59600) 2024-02-19T21:55:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.082+0000 7fe9dd7a5700 1 -- 172.21.15.173:0/1095446719 learned_addr learned my addr 172.21.15.173:0/1095446719 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:52.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.082+0000 7fe9dd7a5700 1 -- 172.21.15.173:0/1095446719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d4005040 con 0x7fe9d80f62a0 2024-02-19T21:55:52.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.082+0000 7fe9dd7a5700 1 --2- 172.21.15.173:0/1095446719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d80f62a0 0x7fe9d81065d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe9d40089f0 tx=0x7fe9d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:52.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.082+0000 7fe9cdffb700 1 -- 172.21.15.173:0/1095446719 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9d4009320 con 0x7fe9d80f62a0 2024-02-19T21:55:52.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.082+0000 7fe9cdffb700 1 -- 172.21.15.173:0/1095446719 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9d4013380 con 0x7fe9d80f62a0 2024-02-19T21:55:52.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.082+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1095446719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d8106cc0 con 0x7fe9d80f62a0 2024-02-19T21:55:52.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.082+0000 7fe9cdffb700 1 -- 172.21.15.173:0/1095446719 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9d4014e20 con 0x7fe9d80f62a0 2024-02-19T21:55:52.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.083+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1095446719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d8107150 con 0x7fe9d80f62a0 2024-02-19T21:55:52.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.083+0000 7fe9cdffb700 1 -- 172.21.15.173:0/1095446719 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9d40095f0 con 0x7fe9d80f62a0 2024-02-19T21:55:52.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.083+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1095446719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d8045ad0 con 0x7fe9d80f62a0 2024-02-19T21:55:52.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.084+0000 7fe9cdffb700 1 --2- 172.21.15.173:0/1095446719 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe9c402ff10 0x7fe9c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:52.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.084+0000 7fe9cdffb700 1 -- 172.21.15.173:0/1095446719 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9d4040ec0 con 0x7fe9d80f62a0 2024-02-19T21:55:52.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.084+0000 7fe9dcfa4700 1 --2- 172.21.15.173:0/1095446719 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe9c402ff10 0x7fe9c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:52.085 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.084+0000 7fe9dcfa4700 1 --2- 172.21.15.173:0/1095446719 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe9c402ff10 0x7fe9c40323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe9c80096a0 tx=0x7fe9c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:52.088 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.088+0000 7fe9cdffb700 1 -- 172.21.15.173:0/1095446719 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9d4014420 con 0x7fe9d80f62a0 2024-02-19T21:55:52.297 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.297+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1095446719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9d8107fa0 con 0x7fe9d80f62a0 2024-02-19T21:55:52.297 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.298+0000 7fe9cdffb700 1 -- 172.21.15.173:0/1095446719 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe9d4010230 con 0x7fe9d80f62a0 2024-02-19T21:55:52.298 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:52.298 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:52.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.299+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1095446719 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe9c402ff10 msgr2=0x7fe9c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:52.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.299+0000 7fe9de7a7700 1 --2- 172.21.15.173:0/1095446719 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe9c402ff10 0x7fe9c40323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe9c80096a0 tx=0x7fe9c8006b40).stop 2024-02-19T21:55:52.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.299+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1095446719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d80f62a0 msgr2=0x7fe9d81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:52.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.299+0000 7fe9de7a7700 1 --2- 172.21.15.173:0/1095446719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d80f62a0 0x7fe9d81065d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe9d40089f0 tx=0x7fe9d400b040).stop 2024-02-19T21:55:52.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.299+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1095446719 shutdown_connections 2024-02-19T21:55:52.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.299+0000 7fe9de7a7700 1 --2- 172.21.15.173:0/1095446719 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe9c402ff10 0x7fe9c40323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:52.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.299+0000 7fe9de7a7700 1 --2- 172.21.15.173:0/1095446719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d80f62a0 0x7fe9d81065d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:52.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.299+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1095446719 >> 172.21.15.173:0/1095446719 conn(0x7fe9d80f1aa0 msgr2=0x7fe9d80f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:52.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.299+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1095446719 shutdown_connections 2024-02-19T21:55:52.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:52.299+0000 7fe9de7a7700 1 -- 172.21.15.173:0/1095446719 wait complete. 2024-02-19T21:55:52.300 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:52.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:52.299001+0000 mon.smithi143 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.173:0/1095446719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:53.868 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:53.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:52.453290+0000 mgr.smithi143.bissbc (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:55.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.754+0000 7f8e59d03700 1 -- 172.21.15.173:0/207857742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e540f5410 msgr2=0x7f8e540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:55.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.754+0000 7f8e59d03700 1 --2- 172.21.15.173:0/207857742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e540f5410 0x7f8e540f5830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8e48009fb0 tx=0x7f8e48009b30).stop 2024-02-19T21:55:55.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.754+0000 7f8e59d03700 1 -- 172.21.15.173:0/207857742 shutdown_connections 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.754+0000 7f8e59d03700 1 --2- 172.21.15.173:0/207857742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e540f5410 0x7f8e540f5830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.754+0000 7f8e59d03700 1 -- 172.21.15.173:0/207857742 >> 172.21.15.173:0/207857742 conn(0x7f8e540f0b70 msgr2=0x7f8e540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.754+0000 7f8e59d03700 1 -- 172.21.15.173:0/207857742 shutdown_connections 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.754+0000 7f8e59d03700 1 -- 172.21.15.173:0/207857742 wait complete. 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.755+0000 7f8e59d03700 1 Processor -- start 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.755+0000 7f8e59d03700 1 -- start start 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.755+0000 7f8e59d03700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e540f5410 0x7f8e54106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.755+0000 7f8e59d03700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e54106cf0 con 0x7f8e540f5410 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.755+0000 7f8e537fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e540f5410 0x7f8e54106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.755+0000 7f8e537fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e540f5410 0x7f8e54106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:59626/0 (socket says 172.21.15.173:59626) 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.755+0000 7f8e537fe700 1 -- 172.21.15.173:0/722876243 learned_addr learned my addr 172.21.15.173:0/722876243 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.755+0000 7f8e537fe700 1 -- 172.21.15.173:0/722876243 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e48005040 con 0x7f8e540f5410 2024-02-19T21:55:55.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.756+0000 7f8e537fe700 1 --2- 172.21.15.173:0/722876243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e540f5410 0x7f8e54106800 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8e48005010 tx=0x7f8e4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:55.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.756+0000 7f8e3bfff700 1 -- 172.21.15.173:0/722876243 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e48009320 con 0x7f8e540f5410 2024-02-19T21:55:55.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.756+0000 7f8e59d03700 1 -- 172.21.15.173:0/722876243 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e54106ef0 con 0x7f8e540f5410 2024-02-19T21:55:55.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.756+0000 7f8e3bfff700 1 -- 172.21.15.173:0/722876243 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e48013380 con 0x7f8e540f5410 2024-02-19T21:55:55.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.756+0000 7f8e3bfff700 1 -- 172.21.15.173:0/722876243 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e48014e50 con 0x7f8e540f5410 2024-02-19T21:55:55.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.756+0000 7f8e59d03700 1 -- 172.21.15.173:0/722876243 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e54107380 con 0x7f8e540f5410 2024-02-19T21:55:55.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.757+0000 7f8e3bfff700 1 -- 172.21.15.173:0/722876243 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e48014420 con 0x7f8e540f5410 2024-02-19T21:55:55.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.757+0000 7f8e59d03700 1 -- 172.21.15.173:0/722876243 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e54045ad0 con 0x7f8e540f5410 2024-02-19T21:55:55.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.757+0000 7f8e3bfff700 1 --2- 172.21.15.173:0/722876243 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8e3c02ff10 0x7f8e3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:55.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.757+0000 7f8e3bfff700 1 -- 172.21.15.173:0/722876243 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e48040e30 con 0x7f8e540f5410 2024-02-19T21:55:55.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.757+0000 7f8e52ffd700 1 --2- 172.21.15.173:0/722876243 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8e3c02ff10 0x7f8e3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:55.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.757+0000 7f8e52ffd700 1 --2- 172.21.15.173:0/722876243 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8e3c02ff10 0x7f8e3c0323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8e440096a0 tx=0x7f8e44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:55.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.762+0000 7f8e3bfff700 1 -- 172.21.15.173:0/722876243 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e480095f0 con 0x7f8e540f5410 2024-02-19T21:55:55.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.964+0000 7f8e59d03700 1 -- 172.21.15.173:0/722876243 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e54108230 con 0x7f8e540f5410 2024-02-19T21:55:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.965+0000 7f8e3bfff700 1 -- 172.21.15.173:0/722876243 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8e4801db70 con 0x7f8e540f5410 2024-02-19T21:55:55.968 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:55.968 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:55.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.966+0000 7f8e59d03700 1 -- 172.21.15.173:0/722876243 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8e3c02ff10 msgr2=0x7f8e3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:55.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.966+0000 7f8e59d03700 1 --2- 172.21.15.173:0/722876243 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8e3c02ff10 0x7f8e3c0323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8e440096a0 tx=0x7f8e44006b40).stop 2024-02-19T21:55:55.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.966+0000 7f8e59d03700 1 -- 172.21.15.173:0/722876243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e540f5410 msgr2=0x7f8e54106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:55.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.966+0000 7f8e59d03700 1 --2- 172.21.15.173:0/722876243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e540f5410 0x7f8e54106800 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8e48005010 tx=0x7f8e4800b040).stop 2024-02-19T21:55:55.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.967+0000 7f8e59d03700 1 -- 172.21.15.173:0/722876243 shutdown_connections 2024-02-19T21:55:55.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.967+0000 7f8e59d03700 1 --2- 172.21.15.173:0/722876243 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8e3c02ff10 0x7f8e3c0323d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:55.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.967+0000 7f8e59d03700 1 --2- 172.21.15.173:0/722876243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e540f5410 0x7f8e54106800 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:55.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.967+0000 7f8e59d03700 1 -- 172.21.15.173:0/722876243 >> 172.21.15.173:0/722876243 conn(0x7f8e540f0b70 msgr2=0x7f8e5405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:55.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.967+0000 7f8e59d03700 1 -- 172.21.15.173:0/722876243 shutdown_connections 2024-02-19T21:55:55.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:55.967+0000 7f8e59d03700 1 -- 172.21.15.173:0/722876243 wait complete. 2024-02-19T21:55:55.970 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:54.453602+0000 mgr.smithi143.bissbc (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:55.966106+0000 mon.smithi143 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.173:0/722876243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:55:57.654 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:55:57.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:55:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:56.453922+0000 mgr.smithi143.bissbc (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:55:59.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.673+0000 7f3a28fe3700 1 -- 172.21.15.173:0/2070732025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a240601e0 msgr2=0x7f3a240f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:59.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.673+0000 7f3a28fe3700 1 --2- 172.21.15.173:0/2070732025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a240601e0 0x7f3a240f8f20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f3a0c009fb0 tx=0x7f3a0c009b30).stop 2024-02-19T21:55:59.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.673+0000 7f3a28fe3700 1 -- 172.21.15.173:0/2070732025 shutdown_connections 2024-02-19T21:55:59.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.673+0000 7f3a28fe3700 1 --2- 172.21.15.173:0/2070732025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a240601e0 0x7f3a240f8f20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:59.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.673+0000 7f3a28fe3700 1 -- 172.21.15.173:0/2070732025 >> 172.21.15.173:0/2070732025 conn(0x7f3a240f09a0 msgr2=0x7f3a240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:59.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.673+0000 7f3a28fe3700 1 -- 172.21.15.173:0/2070732025 shutdown_connections 2024-02-19T21:55:59.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.673+0000 7f3a28fe3700 1 -- 172.21.15.173:0/2070732025 wait complete. 2024-02-19T21:55:59.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.674+0000 7f3a28fe3700 1 Processor -- start 2024-02-19T21:55:59.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.674+0000 7f3a28fe3700 1 -- start start 2024-02-19T21:55:59.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.674+0000 7f3a28fe3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a240601e0 0x7f3a240f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:59.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.674+0000 7f3a28fe3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240f4e20 con 0x7f3a240601e0 2024-02-19T21:55:59.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.674+0000 7f3a237fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a240601e0 0x7f3a240f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:59.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.674+0000 7f3a237fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a240601e0 0x7f3a240f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:59638/0 (socket says 172.21.15.173:59638) 2024-02-19T21:55:59.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.674+0000 7f3a237fe700 1 -- 172.21.15.173:0/3571013269 learned_addr learned my addr 172.21.15.173:0/3571013269 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:55:59.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.675+0000 7f3a237fe700 1 -- 172.21.15.173:0/3571013269 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c005040 con 0x7f3a240601e0 2024-02-19T21:55:59.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.675+0000 7f3a237fe700 1 --2- 172.21.15.173:0/3571013269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a240601e0 0x7f3a240f69a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f3a0c005010 tx=0x7f3a0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:59.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.675+0000 7f3a1bfff700 1 -- 172.21.15.173:0/3571013269 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a0c009320 con 0x7f3a240601e0 2024-02-19T21:55:59.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.675+0000 7f3a28fe3700 1 -- 172.21.15.173:0/3571013269 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a240f4fc0 con 0x7f3a240601e0 2024-02-19T21:55:59.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.675+0000 7f3a1bfff700 1 -- 172.21.15.173:0/3571013269 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a0c013380 con 0x7f3a240601e0 2024-02-19T21:55:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.675+0000 7f3a1bfff700 1 -- 172.21.15.173:0/3571013269 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a0c014e50 con 0x7f3a240601e0 2024-02-19T21:55:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.675+0000 7f3a28fe3700 1 -- 172.21.15.173:0/3571013269 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a240f5450 con 0x7f3a240601e0 2024-02-19T21:55:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.676+0000 7f3a1bfff700 1 -- 172.21.15.173:0/3571013269 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a0c014420 con 0x7f3a240601e0 2024-02-19T21:55:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.676+0000 7f3a28fe3700 1 -- 172.21.15.173:0/3571013269 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a24059100 con 0x7f3a240601e0 2024-02-19T21:55:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.676+0000 7f3a1bfff700 1 --2- 172.21.15.173:0/3571013269 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3a1002ff10 0x7f3a100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.676+0000 7f3a1bfff700 1 -- 172.21.15.173:0/3571013269 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a0c040f40 con 0x7f3a240601e0 2024-02-19T21:55:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.676+0000 7f3a22ffd700 1 --2- 172.21.15.173:0/3571013269 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3a1002ff10 0x7f3a100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:59.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.677+0000 7f3a22ffd700 1 --2- 172.21.15.173:0/3571013269 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3a1002ff10 0x7f3a100323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f3a140096a0 tx=0x7f3a14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:59.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.681+0000 7f3a1bfff700 1 -- 172.21.15.173:0/3571013269 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a0c0095f0 con 0x7f3a240601e0 2024-02-19T21:55:59.888 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.888+0000 7f3a28fe3700 1 -- 172.21.15.173:0/3571013269 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a240fbe10 con 0x7f3a240601e0 2024-02-19T21:55:59.888 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.888+0000 7f3a1bfff700 1 -- 172.21.15.173:0/3571013269 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3a0c01db70 con 0x7f3a240601e0 2024-02-19T21:55:59.889 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:55:59.889 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:55:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.890+0000 7f3a28fe3700 1 -- 172.21.15.173:0/3571013269 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3a1002ff10 msgr2=0x7f3a100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.890+0000 7f3a28fe3700 1 --2- 172.21.15.173:0/3571013269 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3a1002ff10 0x7f3a100323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f3a140096a0 tx=0x7f3a14006b40).stop 2024-02-19T21:55:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.890+0000 7f3a28fe3700 1 -- 172.21.15.173:0/3571013269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a240601e0 msgr2=0x7f3a240f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.890+0000 7f3a28fe3700 1 --2- 172.21.15.173:0/3571013269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a240601e0 0x7f3a240f69a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f3a0c005010 tx=0x7f3a0c00b040).stop 2024-02-19T21:55:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.890+0000 7f3a28fe3700 1 -- 172.21.15.173:0/3571013269 shutdown_connections 2024-02-19T21:55:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.890+0000 7f3a28fe3700 1 --2- 172.21.15.173:0/3571013269 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3a1002ff10 0x7f3a100323d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.890+0000 7f3a28fe3700 1 --2- 172.21.15.173:0/3571013269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a240601e0 0x7f3a240f69a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.890+0000 7f3a28fe3700 1 -- 172.21.15.173:0/3571013269 >> 172.21.15.173:0/3571013269 conn(0x7f3a240f09a0 msgr2=0x7f3a240f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:59.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.890+0000 7f3a28fe3700 1 -- 172.21.15.173:0/3571013269 shutdown_connections 2024-02-19T21:55:59.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:55:59.890+0000 7f3a28fe3700 1 -- 172.21.15.173:0/3571013269 wait complete. 2024-02-19T21:55:59.891 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:55:59.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:55:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:55:58.454132+0000 mgr.smithi143.bissbc (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:55:59.889628+0000 mon.smithi143 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.173:0/3571013269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:01.630 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:01.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:01.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:01 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:00.454364+0000 mgr.smithi143.bissbc (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:03.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.591+0000 7fcce19ca700 1 -- 172.21.15.173:0/1653551290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc0f4b80 msgr2=0x7fccdc0f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:03.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.591+0000 7fcce19ca700 1 --2- 172.21.15.173:0/1653551290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc0f4b80 0x7fccdc0f4fa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fcccc009960 tx=0x7fcccc004d10).stop 2024-02-19T21:56:03.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.591+0000 7fcce19ca700 1 -- 172.21.15.173:0/1653551290 shutdown_connections 2024-02-19T21:56:03.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.591+0000 7fcce19ca700 1 --2- 172.21.15.173:0/1653551290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc0f4b80 0x7fccdc0f4fa0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:03.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.591+0000 7fcce19ca700 1 -- 172.21.15.173:0/1653551290 >> 172.21.15.173:0/1653551290 conn(0x7fccdc0f02e0 msgr2=0x7fccdc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:03.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.591+0000 7fcce19ca700 1 -- 172.21.15.173:0/1653551290 shutdown_connections 2024-02-19T21:56:03.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.591+0000 7fcce19ca700 1 -- 172.21.15.173:0/1653551290 wait complete. 2024-02-19T21:56:03.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.592+0000 7fcce19ca700 1 Processor -- start 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.592+0000 7fcce19ca700 1 -- start start 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.592+0000 7fcce19ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc0f4b80 0x7fccdc105760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.592+0000 7fcce19ca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc105c50 con 0x7fccdc0f4b80 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.592+0000 7fccdaffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc0f4b80 0x7fccdc105760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.592+0000 7fccdaffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc0f4b80 0x7fccdc105760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:60934/0 (socket says 172.21.15.173:60934) 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.592+0000 7fccdaffd700 1 -- 172.21.15.173:0/783737696 learned_addr learned my addr 172.21.15.173:0/783737696 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.592+0000 7fccdaffd700 1 -- 172.21.15.173:0/783737696 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc009640 con 0x7fccdc0f4b80 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.593+0000 7fccdaffd700 1 --2- 172.21.15.173:0/783737696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc0f4b80 0x7fccdc105760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fcccc004a10 tx=0x7fcccc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.593+0000 7fccd37fe700 1 -- 172.21.15.173:0/783737696 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcccc018070 con 0x7fccdc0f4b80 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.593+0000 7fccd37fe700 1 -- 172.21.15.173:0/783737696 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcccc007e70 con 0x7fccdc0f4b80 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.593+0000 7fcce19ca700 1 -- 172.21.15.173:0/783737696 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc105e50 con 0x7fccdc0f4b80 2024-02-19T21:56:03.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.593+0000 7fccd37fe700 1 -- 172.21.15.173:0/783737696 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcccc01d420 con 0x7fccdc0f4b80 2024-02-19T21:56:03.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.593+0000 7fcce19ca700 1 -- 172.21.15.173:0/783737696 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc1062e0 con 0x7fccdc0f4b80 2024-02-19T21:56:03.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.594+0000 7fccd37fe700 1 -- 172.21.15.173:0/783737696 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcccc01d6e0 con 0x7fccdc0f4b80 2024-02-19T21:56:03.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.594+0000 7fcce19ca700 1 -- 172.21.15.173:0/783737696 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccdc045ad0 con 0x7fccdc0f4b80 2024-02-19T21:56:03.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.594+0000 7fccd37fe700 1 --2- 172.21.15.173:0/783737696 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fccc802ff60 0x7fccc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:03.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.594+0000 7fccd37fe700 1 -- 172.21.15.173:0/783737696 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcccc0418a0 con 0x7fccdc0f4b80 2024-02-19T21:56:03.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.594+0000 7fccda7fc700 1 --2- 172.21.15.173:0/783737696 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fccc802ff60 0x7fccc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:03.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.594+0000 7fccda7fc700 1 --2- 172.21.15.173:0/783737696 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fccc802ff60 0x7fccc8032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fccc40096a0 tx=0x7fccc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:03.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.599+0000 7fccd37fe700 1 -- 172.21.15.173:0/783737696 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcccc01dae0 con 0x7fccdc0f4b80 2024-02-19T21:56:03.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.803+0000 7fcce19ca700 1 -- 172.21.15.173:0/783737696 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccdc0fafa0 con 0x7fccdc0f4b80 2024-02-19T21:56:03.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.804+0000 7fccd37fe700 1 -- 172.21.15.173:0/783737696 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcccc01b9c0 con 0x7fccdc0f4b80 2024-02-19T21:56:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:03.804 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:03.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.805+0000 7fcce19ca700 1 -- 172.21.15.173:0/783737696 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fccc802ff60 msgr2=0x7fccc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:03.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.805+0000 7fcce19ca700 1 --2- 172.21.15.173:0/783737696 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fccc802ff60 0x7fccc8032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fccc40096a0 tx=0x7fccc4006b40).stop 2024-02-19T21:56:03.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.805+0000 7fcce19ca700 1 -- 172.21.15.173:0/783737696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc0f4b80 msgr2=0x7fccdc105760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:03.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.805+0000 7fcce19ca700 1 --2- 172.21.15.173:0/783737696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc0f4b80 0x7fccdc105760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fcccc004a10 tx=0x7fcccc004d10).stop 2024-02-19T21:56:03.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.805+0000 7fcce19ca700 1 -- 172.21.15.173:0/783737696 shutdown_connections 2024-02-19T21:56:03.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.805+0000 7fcce19ca700 1 --2- 172.21.15.173:0/783737696 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fccc802ff60 0x7fccc8032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:03.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.805+0000 7fcce19ca700 1 --2- 172.21.15.173:0/783737696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccdc0f4b80 0x7fccdc105760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:03.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.805+0000 7fcce19ca700 1 -- 172.21.15.173:0/783737696 >> 172.21.15.173:0/783737696 conn(0x7fccdc0f02e0 msgr2=0x7fccdc0f0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:03.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.805+0000 7fcce19ca700 1 -- 172.21.15.173:0/783737696 shutdown_connections 2024-02-19T21:56:03.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:03.805+0000 7fcce19ca700 1 -- 172.21.15.173:0/783737696 wait complete. 2024-02-19T21:56:03.806 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:02.454684+0000 mgr.smithi143.bissbc (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:03.804969+0000 mon.smithi143 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.173:0/783737696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:05.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:05.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:04.454988+0000 mgr.smithi143.bissbc (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:07.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.274+0000 7f5f670fb700 1 -- 172.21.15.173:0/1204233058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f600f62a0 msgr2=0x7f5f600f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:07.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.274+0000 7f5f670fb700 1 --2- 172.21.15.173:0/1204233058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f600f62a0 0x7f5f600f66c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f5f5c009fb0 tx=0x7f5f5c009b30).stop 2024-02-19T21:56:07.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.274+0000 7f5f670fb700 1 -- 172.21.15.173:0/1204233058 shutdown_connections 2024-02-19T21:56:07.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.274+0000 7f5f670fb700 1 --2- 172.21.15.173:0/1204233058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f600f62a0 0x7f5f600f66c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:07.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.274+0000 7f5f670fb700 1 -- 172.21.15.173:0/1204233058 >> 172.21.15.173:0/1204233058 conn(0x7f5f600f1aa0 msgr2=0x7f5f600f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:07.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.274+0000 7f5f670fb700 1 -- 172.21.15.173:0/1204233058 shutdown_connections 2024-02-19T21:56:07.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.274+0000 7f5f670fb700 1 -- 172.21.15.173:0/1204233058 wait complete. 2024-02-19T21:56:07.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.274+0000 7f5f670fb700 1 Processor -- start 2024-02-19T21:56:07.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.274+0000 7f5f670fb700 1 -- start start 2024-02-19T21:56:07.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.274+0000 7f5f670fb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f600f62a0 0x7f5f601065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:07.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.274+0000 7f5f670fb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60106ac0 con 0x7f5f600f62a0 2024-02-19T21:56:07.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.275+0000 7f5f660f9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f600f62a0 0x7f5f601065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:07.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.275+0000 7f5f660f9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f600f62a0 0x7f5f601065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:60952/0 (socket says 172.21.15.173:60952) 2024-02-19T21:56:07.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.275+0000 7f5f660f9700 1 -- 172.21.15.173:0/546251561 learned_addr learned my addr 172.21.15.173:0/546251561 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:07.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.275+0000 7f5f660f9700 1 -- 172.21.15.173:0/546251561 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c005040 con 0x7f5f600f62a0 2024-02-19T21:56:07.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.275+0000 7f5f660f9700 1 --2- 172.21.15.173:0/546251561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f600f62a0 0x7f5f601065d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5f5c0089f0 tx=0x7f5f5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:07.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.275+0000 7f5f56ffd700 1 -- 172.21.15.173:0/546251561 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f5c009320 con 0x7f5f600f62a0 2024-02-19T21:56:07.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.275+0000 7f5f670fb700 1 -- 172.21.15.173:0/546251561 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f60106cc0 con 0x7f5f600f62a0 2024-02-19T21:56:07.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.275+0000 7f5f56ffd700 1 -- 172.21.15.173:0/546251561 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f5c013380 con 0x7f5f600f62a0 2024-02-19T21:56:07.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.275+0000 7f5f56ffd700 1 -- 172.21.15.173:0/546251561 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f5c014e20 con 0x7f5f600f62a0 2024-02-19T21:56:07.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.275+0000 7f5f670fb700 1 -- 172.21.15.173:0/546251561 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f60107150 con 0x7f5f600f62a0 2024-02-19T21:56:07.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.276+0000 7f5f56ffd700 1 -- 172.21.15.173:0/546251561 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f5c0263a0 con 0x7f5f600f62a0 2024-02-19T21:56:07.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.276+0000 7f5f670fb700 1 -- 172.21.15.173:0/546251561 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f60045ad0 con 0x7f5f600f62a0 2024-02-19T21:56:07.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.276+0000 7f5f56ffd700 1 --2- 172.21.15.173:0/546251561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5f4c02ff10 0x7f5f4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:07.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.276+0000 7f5f56ffd700 1 -- 172.21.15.173:0/546251561 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f5c040fc0 con 0x7f5f600f62a0 2024-02-19T21:56:07.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.276+0000 7f5f658f8700 1 --2- 172.21.15.173:0/546251561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5f4c02ff10 0x7f5f4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:07.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.277+0000 7f5f658f8700 1 --2- 172.21.15.173:0/546251561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5f4c02ff10 0x7f5f4c0323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f5f500096a0 tx=0x7f5f50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:07.280 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.281+0000 7f5f56ffd700 1 -- 172.21.15.173:0/546251561 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f5c01db70 con 0x7f5f600f62a0 2024-02-19T21:56:07.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.486+0000 7f5f670fb700 1 -- 172.21.15.173:0/546251561 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f60108000 con 0x7f5f600f62a0 2024-02-19T21:56:07.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.487+0000 7f5f56ffd700 1 -- 172.21.15.173:0/546251561 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5f5c01db70 con 0x7f5f600f62a0 2024-02-19T21:56:07.487 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:07.487 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:07.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.489+0000 7f5f670fb700 1 -- 172.21.15.173:0/546251561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5f4c02ff10 msgr2=0x7f5f4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:07.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.489+0000 7f5f670fb700 1 --2- 172.21.15.173:0/546251561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5f4c02ff10 0x7f5f4c0323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f5f500096a0 tx=0x7f5f50006b40).stop 2024-02-19T21:56:07.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.489+0000 7f5f670fb700 1 -- 172.21.15.173:0/546251561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f600f62a0 msgr2=0x7f5f601065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:07.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.489+0000 7f5f670fb700 1 --2- 172.21.15.173:0/546251561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f600f62a0 0x7f5f601065d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5f5c0089f0 tx=0x7f5f5c00b040).stop 2024-02-19T21:56:07.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.489+0000 7f5f670fb700 1 -- 172.21.15.173:0/546251561 shutdown_connections 2024-02-19T21:56:07.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.489+0000 7f5f670fb700 1 --2- 172.21.15.173:0/546251561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5f4c02ff10 0x7f5f4c0323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:07.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.489+0000 7f5f670fb700 1 --2- 172.21.15.173:0/546251561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f600f62a0 0x7f5f601065d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:07.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.489+0000 7f5f670fb700 1 -- 172.21.15.173:0/546251561 >> 172.21.15.173:0/546251561 conn(0x7f5f600f1aa0 msgr2=0x7f5f600f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:07.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.489+0000 7f5f670fb700 1 -- 172.21.15.173:0/546251561 shutdown_connections 2024-02-19T21:56:07.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:07.489+0000 7f5f670fb700 1 -- 172.21.15.173:0/546251561 wait complete. 2024-02-19T21:56:07.490 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:07.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:07 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:06.455252+0000 mgr.smithi143.bissbc (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:07.488025+0000 mon.smithi143 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.173:0/546251561' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:09.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:09.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:08.455545+0000 mgr.smithi143.bissbc (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:11.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.022+0000 7f5b4fe59700 1 -- 172.21.15.173:0/3791641652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b480f6510 msgr2=0x7f5b480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:11.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.022+0000 7f5b4fe59700 1 --2- 172.21.15.173:0/3791641652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b480f6510 0x7f5b480f6930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5b38009fb0 tx=0x7f5b38009b30).stop 2024-02-19T21:56:11.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.023+0000 7f5b4fe59700 1 -- 172.21.15.173:0/3791641652 shutdown_connections 2024-02-19T21:56:11.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.023+0000 7f5b4fe59700 1 --2- 172.21.15.173:0/3791641652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b480f6510 0x7f5b480f6930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:11.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.023+0000 7f5b4fe59700 1 -- 172.21.15.173:0/3791641652 >> 172.21.15.173:0/3791641652 conn(0x7f5b480f1c70 msgr2=0x7f5b480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:11.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.023+0000 7f5b4fe59700 1 -- 172.21.15.173:0/3791641652 shutdown_connections 2024-02-19T21:56:11.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.023+0000 7f5b4fe59700 1 -- 172.21.15.173:0/3791641652 wait complete. 2024-02-19T21:56:11.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.023+0000 7f5b4fe59700 1 Processor -- start 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.023+0000 7f5b4fe59700 1 -- start start 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.023+0000 7f5b4fe59700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b480f6510 0x7f5b4806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.023+0000 7f5b4fe59700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b48069e20 con 0x7f5b480f6510 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.023+0000 7f5b4dbf5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b480f6510 0x7f5b4806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.024+0000 7f5b4dbf5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b480f6510 0x7f5b4806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:60978/0 (socket says 172.21.15.173:60978) 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.024+0000 7f5b4dbf5700 1 -- 172.21.15.173:0/132896091 learned_addr learned my addr 172.21.15.173:0/132896091 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.024+0000 7f5b4dbf5700 1 -- 172.21.15.173:0/132896091 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b38005040 con 0x7f5b480f6510 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.024+0000 7f5b4dbf5700 1 --2- 172.21.15.173:0/132896091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b480f6510 0x7f5b4806b9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f5b38005010 tx=0x7f5b3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.024+0000 7f5b3e7fc700 1 -- 172.21.15.173:0/132896091 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b38009320 con 0x7f5b480f6510 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.024+0000 7f5b3e7fc700 1 -- 172.21.15.173:0/132896091 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b38013380 con 0x7f5b480f6510 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.024+0000 7f5b4fe59700 1 -- 172.21.15.173:0/132896091 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b48069fc0 con 0x7f5b480f6510 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.024+0000 7f5b3e7fc700 1 -- 172.21.15.173:0/132896091 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b38014e50 con 0x7f5b480f6510 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.024+0000 7f5b4fe59700 1 -- 172.21.15.173:0/132896091 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4806a450 con 0x7f5b480f6510 2024-02-19T21:56:11.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.025+0000 7f5b3e7fc700 1 -- 172.21.15.173:0/132896091 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b38014420 con 0x7f5b480f6510 2024-02-19T21:56:11.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.025+0000 7f5b4fe59700 1 -- 172.21.15.173:0/132896091 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b48059100 con 0x7f5b480f6510 2024-02-19T21:56:11.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.025+0000 7f5b3e7fc700 1 --2- 172.21.15.173:0/132896091 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5b3402ff10 0x7f5b340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:11.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.025+0000 7f5b3e7fc700 1 -- 172.21.15.173:0/132896091 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b38040e60 con 0x7f5b480f6510 2024-02-19T21:56:11.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.025+0000 7f5b4d3f4700 1 --2- 172.21.15.173:0/132896091 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5b3402ff10 0x7f5b340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:11.027 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.026+0000 7f5b4d3f4700 1 --2- 172.21.15.173:0/132896091 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5b3402ff10 0x7f5b340323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f5b440096a0 tx=0x7f5b44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:11.030 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.030+0000 7f5b3e7fc700 1 -- 172.21.15.173:0/132896091 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b380095f0 con 0x7f5b480f6510 2024-02-19T21:56:11.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.235+0000 7f5b4fe59700 1 -- 172.21.15.173:0/132896091 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b480fc0a0 con 0x7f5b480f6510 2024-02-19T21:56:11.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.235+0000 7f5b3e7fc700 1 -- 172.21.15.173:0/132896091 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5b3801db70 con 0x7f5b480f6510 2024-02-19T21:56:11.235 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:11.235 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:11.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.237+0000 7f5b4fe59700 1 -- 172.21.15.173:0/132896091 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5b3402ff10 msgr2=0x7f5b340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:11.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.237+0000 7f5b4fe59700 1 --2- 172.21.15.173:0/132896091 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5b3402ff10 0x7f5b340323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f5b440096a0 tx=0x7f5b44006b40).stop 2024-02-19T21:56:11.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.237+0000 7f5b4fe59700 1 -- 172.21.15.173:0/132896091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b480f6510 msgr2=0x7f5b4806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:11.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.237+0000 7f5b4fe59700 1 --2- 172.21.15.173:0/132896091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b480f6510 0x7f5b4806b9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f5b38005010 tx=0x7f5b3800b040).stop 2024-02-19T21:56:11.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.237+0000 7f5b4fe59700 1 -- 172.21.15.173:0/132896091 shutdown_connections 2024-02-19T21:56:11.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.237+0000 7f5b4fe59700 1 --2- 172.21.15.173:0/132896091 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5b3402ff10 0x7f5b340323d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:11.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.237+0000 7f5b4fe59700 1 --2- 172.21.15.173:0/132896091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b480f6510 0x7f5b4806b9a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:11.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.237+0000 7f5b4fe59700 1 -- 172.21.15.173:0/132896091 >> 172.21.15.173:0/132896091 conn(0x7f5b480f1c70 msgr2=0x7f5b480f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:11.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.237+0000 7f5b4fe59700 1 -- 172.21.15.173:0/132896091 shutdown_connections 2024-02-19T21:56:11.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:11.237+0000 7f5b4fe59700 1 -- 172.21.15.173:0/132896091 wait complete. 2024-02-19T21:56:11.238 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:11.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:11 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:11.236626+0000 mon.smithi143 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.173:0/132896091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:12.848 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:12.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:12.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:10.455838+0000 mgr.smithi143.bissbc (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:14.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.705+0000 7f9e05a56700 1 -- 172.21.15.173:0/1730692580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e000f5410 msgr2=0x7f9e000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:14.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.705+0000 7f9e05a56700 1 --2- 172.21.15.173:0/1730692580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e000f5410 0x7f9e000f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f9df4009fb0 tx=0x7f9df4009b30).stop 2024-02-19T21:56:14.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.705+0000 7f9e05a56700 1 -- 172.21.15.173:0/1730692580 shutdown_connections 2024-02-19T21:56:14.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.705+0000 7f9e05a56700 1 --2- 172.21.15.173:0/1730692580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e000f5410 0x7f9e000f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:14.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.705+0000 7f9e05a56700 1 -- 172.21.15.173:0/1730692580 >> 172.21.15.173:0/1730692580 conn(0x7f9e000f0b70 msgr2=0x7f9e000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:14.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.705+0000 7f9e05a56700 1 -- 172.21.15.173:0/1730692580 shutdown_connections 2024-02-19T21:56:14.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.705+0000 7f9e05a56700 1 -- 172.21.15.173:0/1730692580 wait complete. 2024-02-19T21:56:14.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.706+0000 7f9e05a56700 1 Processor -- start 2024-02-19T21:56:14.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.706+0000 7f9e05a56700 1 -- start start 2024-02-19T21:56:14.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.706+0000 7f9e05a56700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e000f5410 0x7f9e001056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:14.748 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.706+0000 7f9e05a56700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e00105bd0 con 0x7f9e000f5410 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.706+0000 7f9dfeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e000f5410 0x7f9e001056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.706+0000 7f9dfeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e000f5410 0x7f9e001056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43434/0 (socket says 172.21.15.173:43434) 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.706+0000 7f9dfeffd700 1 -- 172.21.15.173:0/2571339991 learned_addr learned my addr 172.21.15.173:0/2571339991 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.707+0000 7f9dfeffd700 1 -- 172.21.15.173:0/2571339991 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df4005040 con 0x7f9e000f5410 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.707+0000 7f9dfeffd700 1 --2- 172.21.15.173:0/2571339991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e000f5410 0x7f9e001056e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9df4005010 tx=0x7f9df400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.707+0000 7f9de77fe700 1 -- 172.21.15.173:0/2571339991 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9df4009320 con 0x7f9e000f5410 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.707+0000 7f9de77fe700 1 -- 172.21.15.173:0/2571339991 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9df4013380 con 0x7f9e000f5410 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.707+0000 7f9de77fe700 1 -- 172.21.15.173:0/2571339991 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9df4014e50 con 0x7f9e000f5410 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.707+0000 7f9e05a56700 1 -- 172.21.15.173:0/2571339991 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e00105dd0 con 0x7f9e000f5410 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.707+0000 7f9e05a56700 1 -- 172.21.15.173:0/2571339991 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e00106260 con 0x7f9e000f5410 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.708+0000 7f9de77fe700 1 -- 172.21.15.173:0/2571339991 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9df40095f0 con 0x7f9e000f5410 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.708+0000 7f9e05a56700 1 -- 172.21.15.173:0/2571339991 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e00045ad0 con 0x7f9e000f5410 2024-02-19T21:56:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.708+0000 7f9de77fe700 1 --2- 172.21.15.173:0/2571339991 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9de802ff10 0x7f9de80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:14.750 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.708+0000 7f9de77fe700 1 -- 172.21.15.173:0/2571339991 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9df4040f70 con 0x7f9e000f5410 2024-02-19T21:56:14.750 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.708+0000 7f9dfe7fc700 1 --2- 172.21.15.173:0/2571339991 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9de802ff10 0x7f9de80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:14.750 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.708+0000 7f9dfe7fc700 1 --2- 172.21.15.173:0/2571339991 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9de802ff10 0x7f9de80323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9df00096a0 tx=0x7f9df0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:14.750 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.713+0000 7f9de77fe700 1 -- 172.21.15.173:0/2571339991 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9df40263a0 con 0x7f9e000f5410 2024-02-19T21:56:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:12.456033+0000 mgr.smithi143.bissbc (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:14.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.918+0000 7f9e05a56700 1 -- 172.21.15.173:0/2571339991 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e00107110 con 0x7f9e000f5410 2024-02-19T21:56:14.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.919+0000 7f9de77fe700 1 -- 172.21.15.173:0/2571339991 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9df401d980 con 0x7f9e000f5410 2024-02-19T21:56:14.920 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:14.920 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:14.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.920+0000 7f9e05a56700 1 -- 172.21.15.173:0/2571339991 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9de802ff10 msgr2=0x7f9de80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:14.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.920+0000 7f9e05a56700 1 --2- 172.21.15.173:0/2571339991 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9de802ff10 0x7f9de80323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9df00096a0 tx=0x7f9df0006b40).stop 2024-02-19T21:56:14.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.920+0000 7f9e05a56700 1 -- 172.21.15.173:0/2571339991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e000f5410 msgr2=0x7f9e001056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:14.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.920+0000 7f9e05a56700 1 --2- 172.21.15.173:0/2571339991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e000f5410 0x7f9e001056e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9df4005010 tx=0x7f9df400b040).stop 2024-02-19T21:56:14.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.920+0000 7f9e05a56700 1 -- 172.21.15.173:0/2571339991 shutdown_connections 2024-02-19T21:56:14.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.920+0000 7f9e05a56700 1 --2- 172.21.15.173:0/2571339991 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9de802ff10 0x7f9de80323d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:14.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.920+0000 7f9e05a56700 1 --2- 172.21.15.173:0/2571339991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e000f5410 0x7f9e001056e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:14.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.920+0000 7f9e05a56700 1 -- 172.21.15.173:0/2571339991 >> 172.21.15.173:0/2571339991 conn(0x7f9e000f0b70 msgr2=0x7f9e000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:14.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.920+0000 7f9e05a56700 1 -- 172.21.15.173:0/2571339991 shutdown_connections 2024-02-19T21:56:14.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:14.920+0000 7f9e05a56700 1 -- 172.21.15.173:0/2571339991 wait complete. 2024-02-19T21:56:14.922 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:15.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:15 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:14.920984+0000 mon.smithi143 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.173:0/2571339991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:16.676 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:16.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:14.456302+0000 mgr.smithi143.bissbc (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:18.550 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.543+0000 7fe707135700 1 -- 172.21.15.173:0/2178213052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7000f51a0 msgr2=0x7fe7000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:18.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.543+0000 7fe707135700 1 --2- 172.21.15.173:0/2178213052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7000f51a0 0x7fe7000f55c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe6f0009fb0 tx=0x7fe6f0009b30).stop 2024-02-19T21:56:18.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.543+0000 7fe707135700 1 -- 172.21.15.173:0/2178213052 shutdown_connections 2024-02-19T21:56:18.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.543+0000 7fe707135700 1 --2- 172.21.15.173:0/2178213052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7000f51a0 0x7fe7000f55c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:18.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.543+0000 7fe707135700 1 -- 172.21.15.173:0/2178213052 >> 172.21.15.173:0/2178213052 conn(0x7fe7000f09a0 msgr2=0x7fe7000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:18.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.543+0000 7fe707135700 1 -- 172.21.15.173:0/2178213052 shutdown_connections 2024-02-19T21:56:18.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.543+0000 7fe707135700 1 -- 172.21.15.173:0/2178213052 wait complete. 2024-02-19T21:56:18.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.543+0000 7fe707135700 1 Processor -- start 2024-02-19T21:56:18.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.543+0000 7fe707135700 1 -- start start 2024-02-19T21:56:18.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.544+0000 7fe707135700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7000f51a0 0x7fe7001054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:18.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.544+0000 7fe707135700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7001059c0 con 0x7fe7000f51a0 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.544+0000 7fe706133700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7000f51a0 0x7fe7001054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.544+0000 7fe706133700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7000f51a0 0x7fe7001054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43444/0 (socket says 172.21.15.173:43444) 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.544+0000 7fe706133700 1 -- 172.21.15.173:0/1453093553 learned_addr learned my addr 172.21.15.173:0/1453093553 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.544+0000 7fe706133700 1 -- 172.21.15.173:0/1453093553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f0005040 con 0x7fe7000f51a0 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.544+0000 7fe706133700 1 --2- 172.21.15.173:0/1453093553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7000f51a0 0x7fe7001054d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe6f00089f0 tx=0x7fe6f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.544+0000 7fe6f6ffd700 1 -- 172.21.15.173:0/1453093553 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6f0009320 con 0x7fe7000f51a0 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.545+0000 7fe707135700 1 -- 172.21.15.173:0/1453093553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe700105bc0 con 0x7fe7000f51a0 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.545+0000 7fe6f6ffd700 1 -- 172.21.15.173:0/1453093553 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6f0013380 con 0x7fe7000f51a0 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.545+0000 7fe6f6ffd700 1 -- 172.21.15.173:0/1453093553 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6f0014e20 con 0x7fe7000f51a0 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.545+0000 7fe707135700 1 -- 172.21.15.173:0/1453093553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe700106050 con 0x7fe7000f51a0 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.545+0000 7fe707135700 1 -- 172.21.15.173:0/1453093553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe700045ad0 con 0x7fe7000f51a0 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.545+0000 7fe6f6ffd700 1 -- 172.21.15.173:0/1453093553 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6f00095f0 con 0x7fe7000f51a0 2024-02-19T21:56:18.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.546+0000 7fe6f6ffd700 1 --2- 172.21.15.173:0/1453093553 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe6ec02ff10 0x7fe6ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:18.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.546+0000 7fe6f6ffd700 1 -- 172.21.15.173:0/1453093553 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe6f0040e90 con 0x7fe7000f51a0 2024-02-19T21:56:18.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.546+0000 7fe705932700 1 --2- 172.21.15.173:0/1453093553 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe6ec02ff10 0x7fe6ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:18.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.546+0000 7fe705932700 1 --2- 172.21.15.173:0/1453093553 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe6ec02ff10 0x7fe6ec0323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe6fc0096a0 tx=0x7fe6fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:18.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.551+0000 7fe6f6ffd700 1 -- 172.21.15.173:0/1453093553 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe6f0014420 con 0x7fe7000f51a0 2024-02-19T21:56:18.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.756+0000 7fe707135700 1 -- 172.21.15.173:0/1453093553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe700106f00 con 0x7fe7000f51a0 2024-02-19T21:56:18.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.757+0000 7fe6f6ffd700 1 -- 172.21.15.173:0/1453093553 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe6f001d980 con 0x7fe7000f51a0 2024-02-19T21:56:18.757 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:18.757 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:18.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.758+0000 7fe707135700 1 -- 172.21.15.173:0/1453093553 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe6ec02ff10 msgr2=0x7fe6ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:18.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.758+0000 7fe707135700 1 --2- 172.21.15.173:0/1453093553 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe6ec02ff10 0x7fe6ec0323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe6fc0096a0 tx=0x7fe6fc006b40).stop 2024-02-19T21:56:18.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.758+0000 7fe707135700 1 -- 172.21.15.173:0/1453093553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7000f51a0 msgr2=0x7fe7001054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:18.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.758+0000 7fe707135700 1 --2- 172.21.15.173:0/1453093553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7000f51a0 0x7fe7001054d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe6f00089f0 tx=0x7fe6f000b040).stop 2024-02-19T21:56:18.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.758+0000 7fe707135700 1 -- 172.21.15.173:0/1453093553 shutdown_connections 2024-02-19T21:56:18.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.758+0000 7fe707135700 1 --2- 172.21.15.173:0/1453093553 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe6ec02ff10 0x7fe6ec0323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:18.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.758+0000 7fe707135700 1 --2- 172.21.15.173:0/1453093553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7000f51a0 0x7fe7001054d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:18.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.758+0000 7fe707135700 1 -- 172.21.15.173:0/1453093553 >> 172.21.15.173:0/1453093553 conn(0x7fe7000f09a0 msgr2=0x7fe7000f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:18.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.758+0000 7fe707135700 1 -- 172.21.15.173:0/1453093553 shutdown_connections 2024-02-19T21:56:18.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:18.758+0000 7fe707135700 1 -- 172.21.15.173:0/1453093553 wait complete. 2024-02-19T21:56:18.759 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:16.456623+0000 mgr.smithi143.bissbc (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:19.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:19 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:18.757982+0000 mon.smithi143 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.173:0/1453093553' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:20.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:20.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:18.457013+0000 mgr.smithi143.bissbc (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:20.561541+0000 mon.smithi143 (mon.0) 141 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:56:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:20.562263+0000 mon.smithi143 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T21:56:22.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.273+0000 7f9c79098700 1 -- 172.21.15.173:0/662686149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c740f47c0 msgr2=0x7f9c740f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:22.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.273+0000 7f9c79098700 1 --2- 172.21.15.173:0/662686149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c740f47c0 0x7f9c740f4be0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f9c64009fb0 tx=0x7f9c64009b30).stop 2024-02-19T21:56:22.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.273+0000 7f9c79098700 1 -- 172.21.15.173:0/662686149 shutdown_connections 2024-02-19T21:56:22.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.273+0000 7f9c79098700 1 --2- 172.21.15.173:0/662686149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c740f47c0 0x7f9c740f4be0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:22.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.273+0000 7f9c79098700 1 -- 172.21.15.173:0/662686149 >> 172.21.15.173:0/662686149 conn(0x7f9c740f02e0 msgr2=0x7f9c740f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:22.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.273+0000 7f9c79098700 1 -- 172.21.15.173:0/662686149 shutdown_connections 2024-02-19T21:56:22.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.273+0000 7f9c79098700 1 -- 172.21.15.173:0/662686149 wait complete. 2024-02-19T21:56:22.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.273+0000 7f9c79098700 1 Processor -- start 2024-02-19T21:56:22.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.273+0000 7f9c79098700 1 -- start start 2024-02-19T21:56:22.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.273+0000 7f9c79098700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c740f47c0 0x7f9c741067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:22.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.273+0000 7f9c79098700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c74106cc0 con 0x7f9c740f47c0 2024-02-19T21:56:22.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.274+0000 7f9c72d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c740f47c0 0x7f9c741067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:22.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.274+0000 7f9c72d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c740f47c0 0x7f9c741067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:34194/0 (socket says 172.21.15.173:34194) 2024-02-19T21:56:22.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.274+0000 7f9c72d9d700 1 -- 172.21.15.173:0/2853754296 learned_addr learned my addr 172.21.15.173:0/2853754296 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:22.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.274+0000 7f9c72d9d700 1 -- 172.21.15.173:0/2853754296 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c64005040 con 0x7f9c740f47c0 2024-02-19T21:56:22.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.274+0000 7f9c72d9d700 1 --2- 172.21.15.173:0/2853754296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c740f47c0 0x7f9c741067d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9c64005010 tx=0x7f9c6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:22.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.274+0000 7f9c5b7fe700 1 -- 172.21.15.173:0/2853754296 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c64009320 con 0x7f9c740f47c0 2024-02-19T21:56:22.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.274+0000 7f9c5b7fe700 1 -- 172.21.15.173:0/2853754296 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c64013380 con 0x7f9c740f47c0 2024-02-19T21:56:22.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.274+0000 7f9c79098700 1 -- 172.21.15.173:0/2853754296 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c74106ec0 con 0x7f9c740f47c0 2024-02-19T21:56:22.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.274+0000 7f9c5b7fe700 1 -- 172.21.15.173:0/2853754296 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c64014e50 con 0x7f9c740f47c0 2024-02-19T21:56:22.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.275+0000 7f9c79098700 1 -- 172.21.15.173:0/2853754296 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c74107350 con 0x7f9c740f47c0 2024-02-19T21:56:22.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.275+0000 7f9c5b7fe700 1 -- 172.21.15.173:0/2853754296 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c64014420 con 0x7f9c740f47c0 2024-02-19T21:56:22.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.275+0000 7f9c79098700 1 -- 172.21.15.173:0/2853754296 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c74045ad0 con 0x7f9c740f47c0 2024-02-19T21:56:22.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.275+0000 7f9c5b7fe700 1 --2- 172.21.15.173:0/2853754296 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9c5c02ff10 0x7f9c5c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:22.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.275+0000 7f9c5b7fe700 1 -- 172.21.15.173:0/2853754296 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c64040f40 con 0x7f9c740f47c0 2024-02-19T21:56:22.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.276+0000 7f9c7259c700 1 --2- 172.21.15.173:0/2853754296 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9c5c02ff10 0x7f9c5c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:22.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.276+0000 7f9c7259c700 1 --2- 172.21.15.173:0/2853754296 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9c5c02ff10 0x7f9c5c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f9c680096a0 tx=0x7f9c68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:22.280 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.280+0000 7f9c5b7fe700 1 -- 172.21.15.173:0/2853754296 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c640095f0 con 0x7f9c740f47c0 2024-02-19T21:56:22.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.484+0000 7f9c79098700 1 -- 172.21.15.173:0/2853754296 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c74108200 con 0x7f9c740f47c0 2024-02-19T21:56:22.485 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.485+0000 7f9c5b7fe700 1 -- 172.21.15.173:0/2853754296 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9c6401db70 con 0x7f9c740f47c0 2024-02-19T21:56:22.485 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:22.485 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:22.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.487+0000 7f9c79098700 1 -- 172.21.15.173:0/2853754296 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9c5c02ff10 msgr2=0x7f9c5c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:22.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.487+0000 7f9c79098700 1 --2- 172.21.15.173:0/2853754296 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9c5c02ff10 0x7f9c5c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f9c680096a0 tx=0x7f9c68006b40).stop 2024-02-19T21:56:22.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.487+0000 7f9c79098700 1 -- 172.21.15.173:0/2853754296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c740f47c0 msgr2=0x7f9c741067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:22.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.487+0000 7f9c79098700 1 --2- 172.21.15.173:0/2853754296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c740f47c0 0x7f9c741067d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9c64005010 tx=0x7f9c6400b040).stop 2024-02-19T21:56:22.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.487+0000 7f9c79098700 1 -- 172.21.15.173:0/2853754296 shutdown_connections 2024-02-19T21:56:22.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.487+0000 7f9c79098700 1 --2- 172.21.15.173:0/2853754296 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9c5c02ff10 0x7f9c5c0323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:22.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.487+0000 7f9c79098700 1 --2- 172.21.15.173:0/2853754296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c740f47c0 0x7f9c741067d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:22.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.487+0000 7f9c79098700 1 -- 172.21.15.173:0/2853754296 >> 172.21.15.173:0/2853754296 conn(0x7f9c740f02e0 msgr2=0x7f9c740f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:22.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.487+0000 7f9c79098700 1 -- 172.21.15.173:0/2853754296 shutdown_connections 2024-02-19T21:56:22.487 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:22.487+0000 7f9c79098700 1 -- 172.21.15.173:0/2853754296 wait complete. 2024-02-19T21:56:22.488 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:20.457276+0000 mgr.smithi143.bissbc (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:23.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:22.486310+0000 mon.smithi143 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.173:0/2853754296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:24.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:24.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:24.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:22.457568+0000 mgr.smithi143.bissbc (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:25.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.920+0000 7f7805866700 1 -- 172.21.15.173:0/2191075531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78000f62a0 msgr2=0x7f78000f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:25.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.920+0000 7f7805866700 1 --2- 172.21.15.173:0/2191075531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78000f62a0 0x7f78000f66c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f77f4009fb0 tx=0x7f77f4009b30).stop 2024-02-19T21:56:25.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.920+0000 7f7805866700 1 -- 172.21.15.173:0/2191075531 shutdown_connections 2024-02-19T21:56:25.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.920+0000 7f7805866700 1 --2- 172.21.15.173:0/2191075531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78000f62a0 0x7f78000f66c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:25.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.920+0000 7f7805866700 1 -- 172.21.15.173:0/2191075531 >> 172.21.15.173:0/2191075531 conn(0x7f78000f1aa0 msgr2=0x7f78000f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:25.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.920+0000 7f7805866700 1 -- 172.21.15.173:0/2191075531 shutdown_connections 2024-02-19T21:56:25.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.920+0000 7f7805866700 1 -- 172.21.15.173:0/2191075531 wait complete. 2024-02-19T21:56:25.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.920+0000 7f7805866700 1 Processor -- start 2024-02-19T21:56:25.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.920+0000 7f7805866700 1 -- start start 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.921+0000 7f7805866700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78000f62a0 0x7f78001065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.921+0000 7f7805866700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7800106ac0 con 0x7f78000f62a0 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.921+0000 7f7804864700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78000f62a0 0x7f78001065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.921+0000 7f7804864700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78000f62a0 0x7f78001065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:34216/0 (socket says 172.21.15.173:34216) 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.921+0000 7f7804864700 1 -- 172.21.15.173:0/2718008345 learned_addr learned my addr 172.21.15.173:0/2718008345 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.922+0000 7f7804864700 1 -- 172.21.15.173:0/2718008345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f4005040 con 0x7f78000f62a0 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.922+0000 7f7804864700 1 --2- 172.21.15.173:0/2718008345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78000f62a0 0x7f78001065d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f77f40089f0 tx=0x7f77f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.922+0000 7f77fd7fa700 1 -- 172.21.15.173:0/2718008345 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77f4009320 con 0x7f78000f62a0 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.922+0000 7f7805866700 1 -- 172.21.15.173:0/2718008345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7800106cc0 con 0x7f78000f62a0 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.922+0000 7f77fd7fa700 1 -- 172.21.15.173:0/2718008345 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77f4013380 con 0x7f78000f62a0 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.922+0000 7f7805866700 1 -- 172.21.15.173:0/2718008345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7800107150 con 0x7f78000f62a0 2024-02-19T21:56:25.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.922+0000 7f77fd7fa700 1 -- 172.21.15.173:0/2718008345 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77f4014e20 con 0x7f78000f62a0 2024-02-19T21:56:25.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.923+0000 7f77fd7fa700 1 -- 172.21.15.173:0/2718008345 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77f4014420 con 0x7f78000f62a0 2024-02-19T21:56:25.923 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.923+0000 7f7805866700 1 -- 172.21.15.173:0/2718008345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7800045ad0 con 0x7f78000f62a0 2024-02-19T21:56:25.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.923+0000 7f77fd7fa700 1 --2- 172.21.15.173:0/2718008345 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f77f002ff10 0x7f77f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:25.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.923+0000 7f77fd7fa700 1 -- 172.21.15.173:0/2718008345 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f77f4040f70 con 0x7f78000f62a0 2024-02-19T21:56:25.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.923+0000 7f77fffff700 1 --2- 172.21.15.173:0/2718008345 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f77f002ff10 0x7f77f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:25.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.923+0000 7f77fffff700 1 --2- 172.21.15.173:0/2718008345 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f77f002ff10 0x7f77f00323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f77ec0096a0 tx=0x7f77ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:25.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:25.928+0000 7f77fd7fa700 1 -- 172.21.15.173:0/2718008345 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77f4014790 con 0x7f78000f62a0 2024-02-19T21:56:26.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.134+0000 7f7805866700 1 -- 172.21.15.173:0/2718008345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7800107fa0 con 0x7f78000f62a0 2024-02-19T21:56:26.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.135+0000 7f77fd7fa700 1 -- 172.21.15.173:0/2718008345 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f77f401d7f0 con 0x7f78000f62a0 2024-02-19T21:56:26.135 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:26.135 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:26.137 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.136+0000 7f7805866700 1 -- 172.21.15.173:0/2718008345 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f77f002ff10 msgr2=0x7f77f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:26.137 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.136+0000 7f7805866700 1 --2- 172.21.15.173:0/2718008345 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f77f002ff10 0x7f77f00323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f77ec0096a0 tx=0x7f77ec006b40).stop 2024-02-19T21:56:26.137 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.136+0000 7f7805866700 1 -- 172.21.15.173:0/2718008345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78000f62a0 msgr2=0x7f78001065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:26.137 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.136+0000 7f7805866700 1 --2- 172.21.15.173:0/2718008345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78000f62a0 0x7f78001065d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f77f40089f0 tx=0x7f77f400b040).stop 2024-02-19T21:56:26.137 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.136+0000 7f7805866700 1 -- 172.21.15.173:0/2718008345 shutdown_connections 2024-02-19T21:56:26.137 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.136+0000 7f7805866700 1 --2- 172.21.15.173:0/2718008345 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f77f002ff10 0x7f77f00323d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:26.137 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.136+0000 7f7805866700 1 --2- 172.21.15.173:0/2718008345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78000f62a0 0x7f78001065d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:26.138 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.136+0000 7f7805866700 1 -- 172.21.15.173:0/2718008345 >> 172.21.15.173:0/2718008345 conn(0x7f78000f1aa0 msgr2=0x7f78000f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:26.138 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.137+0000 7f7805866700 1 -- 172.21.15.173:0/2718008345 shutdown_connections 2024-02-19T21:56:26.138 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:26.137+0000 7f7805866700 1 -- 172.21.15.173:0/2718008345 wait complete. 2024-02-19T21:56:26.138 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:24.457826+0000 mgr.smithi143.bissbc (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:26.136050+0000 mon.smithi143 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.173:0/2718008345' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:27.809 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:27.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:26.458021+0000 mgr.smithi143.bissbc (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:29.728 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.728+0000 7f5be7fff700 1 -- 172.21.15.173:0/3679693702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be80f51a0 msgr2=0x7f5be80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:29.729 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.728+0000 7f5be7fff700 1 --2- 172.21.15.173:0/3679693702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be80f51a0 0x7f5be80f55c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5bd8009fb0 tx=0x7f5bd8009b30).stop 2024-02-19T21:56:29.729 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.728+0000 7f5be7fff700 1 -- 172.21.15.173:0/3679693702 shutdown_connections 2024-02-19T21:56:29.729 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.728+0000 7f5be7fff700 1 --2- 172.21.15.173:0/3679693702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be80f51a0 0x7f5be80f55c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:29.729 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.728+0000 7f5be7fff700 1 -- 172.21.15.173:0/3679693702 >> 172.21.15.173:0/3679693702 conn(0x7f5be80f09a0 msgr2=0x7f5be80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:29.729 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.728+0000 7f5be7fff700 1 -- 172.21.15.173:0/3679693702 shutdown_connections 2024-02-19T21:56:29.729 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.728+0000 7f5be7fff700 1 -- 172.21.15.173:0/3679693702 wait complete. 2024-02-19T21:56:29.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.729+0000 7f5be7fff700 1 Processor -- start 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.730+0000 7f5be7fff700 1 -- start start 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.730+0000 7f5be7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be80f51a0 0x7f5be81054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.730+0000 7f5be7fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be81059c0 con 0x7f5be80f51a0 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.730+0000 7f5be6ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be80f51a0 0x7f5be81054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.730+0000 7f5be6ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be80f51a0 0x7f5be81054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:34228/0 (socket says 172.21.15.173:34228) 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.730+0000 7f5be6ffd700 1 -- 172.21.15.173:0/4027304854 learned_addr learned my addr 172.21.15.173:0/4027304854 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.730+0000 7f5be6ffd700 1 -- 172.21.15.173:0/4027304854 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd8005040 con 0x7f5be80f51a0 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.731+0000 7f5be6ffd700 1 --2- 172.21.15.173:0/4027304854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be80f51a0 0x7f5be81054d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5bd80089f0 tx=0x7f5bd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.731+0000 7f5bcf7fe700 1 -- 172.21.15.173:0/4027304854 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bd8009320 con 0x7f5be80f51a0 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.731+0000 7f5bcf7fe700 1 -- 172.21.15.173:0/4027304854 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5bd8013380 con 0x7f5be80f51a0 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.731+0000 7f5be7fff700 1 -- 172.21.15.173:0/4027304854 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be8105bc0 con 0x7f5be80f51a0 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.731+0000 7f5bcf7fe700 1 -- 172.21.15.173:0/4027304854 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bd8014e20 con 0x7f5be80f51a0 2024-02-19T21:56:29.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.731+0000 7f5be7fff700 1 -- 172.21.15.173:0/4027304854 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be8106050 con 0x7f5be80f51a0 2024-02-19T21:56:29.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.731+0000 7f5bcf7fe700 1 -- 172.21.15.173:0/4027304854 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5bd80095f0 con 0x7f5be80f51a0 2024-02-19T21:56:29.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.732+0000 7f5be7fff700 1 -- 172.21.15.173:0/4027304854 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be8045ad0 con 0x7f5be80f51a0 2024-02-19T21:56:29.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.732+0000 7f5bcf7fe700 1 --2- 172.21.15.173:0/4027304854 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5bd002ff10 0x7f5bd00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:29.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.732+0000 7f5bcf7fe700 1 -- 172.21.15.173:0/4027304854 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5bd8040e90 con 0x7f5be80f51a0 2024-02-19T21:56:29.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.732+0000 7f5be67fc700 1 --2- 172.21.15.173:0/4027304854 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5bd002ff10 0x7f5bd00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:29.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.732+0000 7f5be67fc700 1 --2- 172.21.15.173:0/4027304854 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5bd002ff10 0x7f5bd00323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f5bdc0096a0 tx=0x7f5bdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:29.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.737+0000 7f5bcf7fe700 1 -- 172.21.15.173:0/4027304854 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bd8014420 con 0x7f5be80f51a0 2024-02-19T21:56:29.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.942+0000 7f5be7fff700 1 -- 172.21.15.173:0/4027304854 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5be8106f00 con 0x7f5be80f51a0 2024-02-19T21:56:29.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.942+0000 7f5bcf7fe700 1 -- 172.21.15.173:0/4027304854 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5bd801d980 con 0x7f5be80f51a0 2024-02-19T21:56:29.942 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:29.943 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:29.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.944+0000 7f5be7fff700 1 -- 172.21.15.173:0/4027304854 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5bd002ff10 msgr2=0x7f5bd00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:29.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.944+0000 7f5be7fff700 1 --2- 172.21.15.173:0/4027304854 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5bd002ff10 0x7f5bd00323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f5bdc0096a0 tx=0x7f5bdc006b40).stop 2024-02-19T21:56:29.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.944+0000 7f5be7fff700 1 -- 172.21.15.173:0/4027304854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be80f51a0 msgr2=0x7f5be81054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:29.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.944+0000 7f5be7fff700 1 --2- 172.21.15.173:0/4027304854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be80f51a0 0x7f5be81054d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5bd80089f0 tx=0x7f5bd800b040).stop 2024-02-19T21:56:29.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.944+0000 7f5be7fff700 1 -- 172.21.15.173:0/4027304854 shutdown_connections 2024-02-19T21:56:29.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.944+0000 7f5be7fff700 1 --2- 172.21.15.173:0/4027304854 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5bd002ff10 0x7f5bd00323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:29.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.944+0000 7f5be7fff700 1 --2- 172.21.15.173:0/4027304854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be80f51a0 0x7f5be81054d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:29.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.944+0000 7f5be7fff700 1 -- 172.21.15.173:0/4027304854 >> 172.21.15.173:0/4027304854 conn(0x7f5be80f09a0 msgr2=0x7f5be80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:29.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.944+0000 7f5be7fff700 1 -- 172.21.15.173:0/4027304854 shutdown_connections 2024-02-19T21:56:29.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:29.944+0000 7f5be7fff700 1 -- 172.21.15.173:0/4027304854 wait complete. 2024-02-19T21:56:29.945 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:28.458280+0000 mgr.smithi143.bissbc (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:29.943336+0000 mon.smithi143 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.173:0/4027304854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:31.504 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:31.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:32.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:30.458557+0000 mgr.smithi143.bissbc (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:33.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.403+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/2301741286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94060220 msgr2=0x7f6d940f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:33.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.403+0000 7f6d9b1c6700 1 --2- 172.21.15.173:0/2301741286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94060220 0x7f6d940f9150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f6d88009fb0 tx=0x7f6d88009b30).stop 2024-02-19T21:56:33.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.403+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/2301741286 shutdown_connections 2024-02-19T21:56:33.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.403+0000 7f6d9b1c6700 1 --2- 172.21.15.173:0/2301741286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94060220 0x7f6d940f9150 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:33.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.403+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/2301741286 >> 172.21.15.173:0/2301741286 conn(0x7f6d940f0b70 msgr2=0x7f6d940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:33.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.403+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/2301741286 shutdown_connections 2024-02-19T21:56:33.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.403+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/2301741286 wait complete. 2024-02-19T21:56:33.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.404+0000 7f6d9b1c6700 1 Processor -- start 2024-02-19T21:56:33.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.404+0000 7f6d9b1c6700 1 -- start start 2024-02-19T21:56:33.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.404+0000 7f6d9b1c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94060220 0x7f6d940f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:33.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.404+0000 7f6d9b1c6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940f5050 con 0x7f6d94060220 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.404+0000 7f6d98f62700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94060220 0x7f6d940f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.404+0000 7f6d98f62700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94060220 0x7f6d940f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41226/0 (socket says 172.21.15.173:41226) 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.404+0000 7f6d98f62700 1 -- 172.21.15.173:0/943100974 learned_addr learned my addr 172.21.15.173:0/943100974 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.404+0000 7f6d98f62700 1 -- 172.21.15.173:0/943100974 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d88005040 con 0x7f6d94060220 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.404+0000 7f6d98f62700 1 --2- 172.21.15.173:0/943100974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94060220 0x7f6d940f6bd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f6d880089f0 tx=0x7f6d8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.405+0000 7f6d917fa700 1 -- 172.21.15.173:0/943100974 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d88009320 con 0x7f6d94060220 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.405+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/943100974 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d940f51f0 con 0x7f6d94060220 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.405+0000 7f6d917fa700 1 -- 172.21.15.173:0/943100974 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d88013380 con 0x7f6d94060220 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.405+0000 7f6d917fa700 1 -- 172.21.15.173:0/943100974 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d88014e20 con 0x7f6d94060220 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.405+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/943100974 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d940f5680 con 0x7f6d94060220 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.406+0000 7f6d917fa700 1 -- 172.21.15.173:0/943100974 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d88014420 con 0x7f6d94060220 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.406+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/943100974 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d94059100 con 0x7f6d94060220 2024-02-19T21:56:33.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.406+0000 7f6d917fa700 1 --2- 172.21.15.173:0/943100974 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6d8402ff10 0x7f6d840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:33.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.406+0000 7f6d917fa700 1 -- 172.21.15.173:0/943100974 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d88040f00 con 0x7f6d94060220 2024-02-19T21:56:33.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.406+0000 7f6d93fff700 1 --2- 172.21.15.173:0/943100974 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6d8402ff10 0x7f6d840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:33.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.406+0000 7f6d93fff700 1 --2- 172.21.15.173:0/943100974 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6d8402ff10 0x7f6d840323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6d800096a0 tx=0x7f6d80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:33.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.411+0000 7f6d917fa700 1 -- 172.21.15.173:0/943100974 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d880095f0 con 0x7f6d94060220 2024-02-19T21:56:33.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.618+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/943100974 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d940fc090 con 0x7f6d94060220 2024-02-19T21:56:33.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.619+0000 7f6d917fa700 1 -- 172.21.15.173:0/943100974 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6d8801db70 con 0x7f6d94060220 2024-02-19T21:56:33.620 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:33.621 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:33.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.620+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/943100974 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6d8402ff10 msgr2=0x7f6d840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:33.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.620+0000 7f6d9b1c6700 1 --2- 172.21.15.173:0/943100974 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6d8402ff10 0x7f6d840323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6d800096a0 tx=0x7f6d80006b40).stop 2024-02-19T21:56:33.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.620+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/943100974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94060220 msgr2=0x7f6d940f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:33.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.620+0000 7f6d9b1c6700 1 --2- 172.21.15.173:0/943100974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94060220 0x7f6d940f6bd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f6d880089f0 tx=0x7f6d8800b040).stop 2024-02-19T21:56:33.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.620+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/943100974 shutdown_connections 2024-02-19T21:56:33.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.620+0000 7f6d9b1c6700 1 --2- 172.21.15.173:0/943100974 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6d8402ff10 0x7f6d840323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:33.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.620+0000 7f6d9b1c6700 1 --2- 172.21.15.173:0/943100974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94060220 0x7f6d940f6bd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:33.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.620+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/943100974 >> 172.21.15.173:0/943100974 conn(0x7f6d940f0b70 msgr2=0x7f6d940f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:33.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.620+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/943100974 shutdown_connections 2024-02-19T21:56:33.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:33.620+0000 7f6d9b1c6700 1 -- 172.21.15.173:0/943100974 wait complete. 2024-02-19T21:56:33.622 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:33.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:33 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:32.458822+0000 mgr.smithi143.bissbc (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:33.620003+0000 mon.smithi143 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.173:0/943100974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:35.268 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:35.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:35.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:34.459103+0000 mgr.smithi143.bissbc (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.137+0000 7f1951d25700 1 -- 172.21.15.173:0/1725311705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f194c060200 msgr2=0x7f194c0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.137+0000 7f1951d25700 1 --2- 172.21.15.173:0/1725311705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f194c060200 0x7f194c0fb570 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f193c009fb0 tx=0x7f193c009b30).stop 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.137+0000 7f1951d25700 1 -- 172.21.15.173:0/1725311705 shutdown_connections 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.137+0000 7f1951d25700 1 --2- 172.21.15.173:0/1725311705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f194c060200 0x7f194c0fb570 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.137+0000 7f1951d25700 1 -- 172.21.15.173:0/1725311705 >> 172.21.15.173:0/1725311705 conn(0x7f194c0f0b70 msgr2=0x7f194c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.137+0000 7f1951d25700 1 -- 172.21.15.173:0/1725311705 shutdown_connections 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.137+0000 7f1951d25700 1 -- 172.21.15.173:0/1725311705 wait complete. 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.138+0000 7f1951d25700 1 Processor -- start 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.138+0000 7f1951d25700 1 -- start start 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.138+0000 7f1951d25700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f194c060200 0x7f194c1089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.138+0000 7f1951d25700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c108e90 con 0x7f194c060200 2024-02-19T21:56:37.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.138+0000 7f194b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f194c060200 0x7f194c1089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:37.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.138+0000 7f194b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f194c060200 0x7f194c1089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41258/0 (socket says 172.21.15.173:41258) 2024-02-19T21:56:37.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.138+0000 7f194b7fe700 1 -- 172.21.15.173:0/3612486249 learned_addr learned my addr 172.21.15.173:0/3612486249 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:37.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.139+0000 7f194b7fe700 1 -- 172.21.15.173:0/3612486249 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c005040 con 0x7f194c060200 2024-02-19T21:56:37.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.139+0000 7f194b7fe700 1 --2- 172.21.15.173:0/3612486249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f194c060200 0x7f194c1089a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f193c0089f0 tx=0x7f193c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:37.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.139+0000 7f1943fff700 1 -- 172.21.15.173:0/3612486249 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f193c009320 con 0x7f194c060200 2024-02-19T21:56:37.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.139+0000 7f1943fff700 1 -- 172.21.15.173:0/3612486249 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f193c013380 con 0x7f194c060200 2024-02-19T21:56:37.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.139+0000 7f1943fff700 1 -- 172.21.15.173:0/3612486249 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f193c014e20 con 0x7f194c060200 2024-02-19T21:56:37.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.139+0000 7f1951d25700 1 -- 172.21.15.173:0/3612486249 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194c109090 con 0x7f194c060200 2024-02-19T21:56:37.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.139+0000 7f1951d25700 1 -- 172.21.15.173:0/3612486249 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194c109520 con 0x7f194c060200 2024-02-19T21:56:37.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.140+0000 7f1943fff700 1 -- 172.21.15.173:0/3612486249 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f193c0095f0 con 0x7f194c060200 2024-02-19T21:56:37.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.140+0000 7f1951d25700 1 -- 172.21.15.173:0/3612486249 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f194c045ad0 con 0x7f194c060200 2024-02-19T21:56:37.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.140+0000 7f1943fff700 1 --2- 172.21.15.173:0/3612486249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f193802ff10 0x7f19380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:37.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.140+0000 7f1943fff700 1 -- 172.21.15.173:0/3612486249 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f193c040f50 con 0x7f194c060200 2024-02-19T21:56:37.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.140+0000 7f194affd700 1 --2- 172.21.15.173:0/3612486249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f193802ff10 0x7f19380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:37.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.140+0000 7f194affd700 1 --2- 172.21.15.173:0/3612486249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f193802ff10 0x7f19380323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f19340096a0 tx=0x7f1934006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:37.144 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.145+0000 7f1943fff700 1 -- 172.21.15.173:0/3612486249 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f193c014420 con 0x7f194c060200 2024-02-19T21:56:37.349 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.349+0000 7f1951d25700 1 -- 172.21.15.173:0/3612486249 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f194c10a3d0 con 0x7f194c060200 2024-02-19T21:56:37.349 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.350+0000 7f1943fff700 1 -- 172.21.15.173:0/3612486249 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f193c01d980 con 0x7f194c060200 2024-02-19T21:56:37.349 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:37.350 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:37.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.351+0000 7f1951d25700 1 -- 172.21.15.173:0/3612486249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f193802ff10 msgr2=0x7f19380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:37.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.351+0000 7f1951d25700 1 --2- 172.21.15.173:0/3612486249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f193802ff10 0x7f19380323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f19340096a0 tx=0x7f1934006b40).stop 2024-02-19T21:56:37.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.351+0000 7f1951d25700 1 -- 172.21.15.173:0/3612486249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f194c060200 msgr2=0x7f194c1089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:37.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.351+0000 7f1951d25700 1 --2- 172.21.15.173:0/3612486249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f194c060200 0x7f194c1089a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f193c0089f0 tx=0x7f193c00b040).stop 2024-02-19T21:56:37.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.351+0000 7f1951d25700 1 -- 172.21.15.173:0/3612486249 shutdown_connections 2024-02-19T21:56:37.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.351+0000 7f1951d25700 1 --2- 172.21.15.173:0/3612486249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f193802ff10 0x7f19380323d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:37.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.351+0000 7f1951d25700 1 --2- 172.21.15.173:0/3612486249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f194c060200 0x7f194c1089a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:37.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.351+0000 7f1951d25700 1 -- 172.21.15.173:0/3612486249 >> 172.21.15.173:0/3612486249 conn(0x7f194c0f0b70 msgr2=0x7f194c0f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:37.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.351+0000 7f1951d25700 1 -- 172.21.15.173:0/3612486249 shutdown_connections 2024-02-19T21:56:37.352 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:37.351+0000 7f1951d25700 1 -- 172.21.15.173:0/3612486249 wait complete. 2024-02-19T21:56:37.352 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:37.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:37 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:36.459438+0000 mgr.smithi143.bissbc (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:37.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:37 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:37.350756+0000 mon.smithi143 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.173:0/3612486249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:38.982 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:38.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:39.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:39 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:38.459727+0000 mgr.smithi143.bissbc (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:40.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.796+0000 7f0c4bf48700 1 -- 172.21.15.173:0/2906246827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440f5410 msgr2=0x7f0c440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:40.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.796+0000 7f0c4bf48700 1 --2- 172.21.15.173:0/2906246827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440f5410 0x7f0c440f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f0c40009fb0 tx=0x7f0c40009b30).stop 2024-02-19T21:56:40.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.796+0000 7f0c4bf48700 1 -- 172.21.15.173:0/2906246827 shutdown_connections 2024-02-19T21:56:40.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.796+0000 7f0c4bf48700 1 --2- 172.21.15.173:0/2906246827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440f5410 0x7f0c440f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:40.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.796+0000 7f0c4bf48700 1 -- 172.21.15.173:0/2906246827 >> 172.21.15.173:0/2906246827 conn(0x7f0c440f0b70 msgr2=0x7f0c440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:40.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.796+0000 7f0c4bf48700 1 -- 172.21.15.173:0/2906246827 shutdown_connections 2024-02-19T21:56:40.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.796+0000 7f0c4bf48700 1 -- 172.21.15.173:0/2906246827 wait complete. 2024-02-19T21:56:40.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.797+0000 7f0c4bf48700 1 Processor -- start 2024-02-19T21:56:40.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.797+0000 7f0c4bf48700 1 -- start start 2024-02-19T21:56:40.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.797+0000 7f0c4bf48700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440f5410 0x7f0c44109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:40.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.797+0000 7f0c4bf48700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c44060220 con 0x7f0c440f5410 2024-02-19T21:56:40.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.797+0000 7f0c49ce4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440f5410 0x7f0c44109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:40.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.797+0000 7f0c49ce4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440f5410 0x7f0c44109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41270/0 (socket says 172.21.15.173:41270) 2024-02-19T21:56:40.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.797+0000 7f0c49ce4700 1 -- 172.21.15.173:0/3826750263 learned_addr learned my addr 172.21.15.173:0/3826750263 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:40.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.797+0000 7f0c49ce4700 1 -- 172.21.15.173:0/3826750263 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c40005040 con 0x7f0c440f5410 2024-02-19T21:56:40.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.798+0000 7f0c49ce4700 1 --2- 172.21.15.173:0/3826750263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440f5410 0x7f0c44109540 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f0c40005010 tx=0x7f0c4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:40.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.798+0000 7f0c3a7fc700 1 -- 172.21.15.173:0/3826750263 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c40009320 con 0x7f0c440f5410 2024-02-19T21:56:40.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.798+0000 7f0c4bf48700 1 -- 172.21.15.173:0/3826750263 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c44060420 con 0x7f0c440f5410 2024-02-19T21:56:40.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.798+0000 7f0c3a7fc700 1 -- 172.21.15.173:0/3826750263 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c40013380 con 0x7f0c440f5410 2024-02-19T21:56:40.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.798+0000 7f0c3a7fc700 1 -- 172.21.15.173:0/3826750263 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c40014e50 con 0x7f0c440f5410 2024-02-19T21:56:40.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.798+0000 7f0c4bf48700 1 -- 172.21.15.173:0/3826750263 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c44109bf0 con 0x7f0c440f5410 2024-02-19T21:56:40.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.799+0000 7f0c3a7fc700 1 -- 172.21.15.173:0/3826750263 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c400095f0 con 0x7f0c440f5410 2024-02-19T21:56:40.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.799+0000 7f0c4bf48700 1 -- 172.21.15.173:0/3826750263 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c44059100 con 0x7f0c440f5410 2024-02-19T21:56:40.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.799+0000 7f0c3a7fc700 1 --2- 172.21.15.173:0/3826750263 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0c3002ff10 0x7f0c300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:40.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.799+0000 7f0c3a7fc700 1 -- 172.21.15.173:0/3826750263 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c40040e90 con 0x7f0c440f5410 2024-02-19T21:56:40.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.799+0000 7f0c494e3700 1 --2- 172.21.15.173:0/3826750263 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0c3002ff10 0x7f0c300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:40.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.799+0000 7f0c494e3700 1 --2- 172.21.15.173:0/3826750263 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0c3002ff10 0x7f0c300323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0c340096a0 tx=0x7f0c34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:40.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:40.804+0000 7f0c3a7fc700 1 -- 172.21.15.173:0/3826750263 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c40014420 con 0x7f0c440f5410 2024-02-19T21:56:41.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.009+0000 7f0c4bf48700 1 -- 172.21.15.173:0/3826750263 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c44045ad0 con 0x7f0c440f5410 2024-02-19T21:56:41.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.010+0000 7f0c3a7fc700 1 -- 172.21.15.173:0/3826750263 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0c40010230 con 0x7f0c440f5410 2024-02-19T21:56:41.010 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:41.010 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:41.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.012+0000 7f0c4bf48700 1 -- 172.21.15.173:0/3826750263 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0c3002ff10 msgr2=0x7f0c300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:41.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.012+0000 7f0c4bf48700 1 --2- 172.21.15.173:0/3826750263 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0c3002ff10 0x7f0c300323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0c340096a0 tx=0x7f0c34006b40).stop 2024-02-19T21:56:41.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.012+0000 7f0c4bf48700 1 -- 172.21.15.173:0/3826750263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440f5410 msgr2=0x7f0c44109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:41.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.012+0000 7f0c4bf48700 1 --2- 172.21.15.173:0/3826750263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440f5410 0x7f0c44109540 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f0c40005010 tx=0x7f0c4000b040).stop 2024-02-19T21:56:41.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.012+0000 7f0c4bf48700 1 -- 172.21.15.173:0/3826750263 shutdown_connections 2024-02-19T21:56:41.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.012+0000 7f0c4bf48700 1 --2- 172.21.15.173:0/3826750263 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0c3002ff10 0x7f0c300323d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:41.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.012+0000 7f0c4bf48700 1 --2- 172.21.15.173:0/3826750263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440f5410 0x7f0c44109540 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:41.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.012+0000 7f0c4bf48700 1 -- 172.21.15.173:0/3826750263 >> 172.21.15.173:0/3826750263 conn(0x7f0c440f0b70 msgr2=0x7f0c440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:41.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.012+0000 7f0c4bf48700 1 -- 172.21.15.173:0/3826750263 shutdown_connections 2024-02-19T21:56:41.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:41.012+0000 7f0c4bf48700 1 -- 172.21.15.173:0/3826750263 wait complete. 2024-02-19T21:56:41.013 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:41.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:41 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:40.460024+0000 mgr.smithi143.bissbc (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:41.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:41 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:41.011129+0000 mon.smithi143 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.173:0/3826750263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:42.705 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:42.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:44.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.686+0000 7f14d68ff700 1 -- 172.21.15.173:0/4227027044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14d00f6510 msgr2=0x7f14d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:44.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.686+0000 7f14d68ff700 1 --2- 172.21.15.173:0/4227027044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14d00f6510 0x7f14d00f6930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f14c0009fb0 tx=0x7f14c0009b30).stop 2024-02-19T21:56:44.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.686+0000 7f14d68ff700 1 -- 172.21.15.173:0/4227027044 shutdown_connections 2024-02-19T21:56:44.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.686+0000 7f14d68ff700 1 --2- 172.21.15.173:0/4227027044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14d00f6510 0x7f14d00f6930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:44.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.686+0000 7f14d68ff700 1 -- 172.21.15.173:0/4227027044 >> 172.21.15.173:0/4227027044 conn(0x7f14d00f1c70 msgr2=0x7f14d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:44.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.686+0000 7f14d68ff700 1 -- 172.21.15.173:0/4227027044 shutdown_connections 2024-02-19T21:56:44.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.686+0000 7f14d68ff700 1 -- 172.21.15.173:0/4227027044 wait complete. 2024-02-19T21:56:44.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.686+0000 7f14d68ff700 1 Processor -- start 2024-02-19T21:56:44.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.687+0000 7f14d68ff700 1 -- start start 2024-02-19T21:56:44.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.687+0000 7f14d68ff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14d00f6510 0x7f14d01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:44.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.687+0000 7f14d68ff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d0106cd0 con 0x7f14d00f6510 2024-02-19T21:56:44.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.687+0000 7f14cffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14d00f6510 0x7f14d01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:44.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.687+0000 7f14cffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14d00f6510 0x7f14d01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:53996/0 (socket says 172.21.15.173:53996) 2024-02-19T21:56:44.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.687+0000 7f14cffff700 1 -- 172.21.15.173:0/3047394701 learned_addr learned my addr 172.21.15.173:0/3047394701 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:44.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.687+0000 7f14cffff700 1 -- 172.21.15.173:0/3047394701 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c0005040 con 0x7f14d00f6510 2024-02-19T21:56:44.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.688+0000 7f14cffff700 1 --2- 172.21.15.173:0/3047394701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14d00f6510 0x7f14d01067e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f14c0005010 tx=0x7f14c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:44.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.688+0000 7f14ccff9700 1 -- 172.21.15.173:0/3047394701 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14c0009320 con 0x7f14d00f6510 2024-02-19T21:56:44.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.688+0000 7f14d68ff700 1 -- 172.21.15.173:0/3047394701 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d0106ed0 con 0x7f14d00f6510 2024-02-19T21:56:44.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.688+0000 7f14ccff9700 1 -- 172.21.15.173:0/3047394701 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14c0013380 con 0x7f14d00f6510 2024-02-19T21:56:44.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.688+0000 7f14ccff9700 1 -- 172.21.15.173:0/3047394701 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14c0014e50 con 0x7f14d00f6510 2024-02-19T21:56:44.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.688+0000 7f14d68ff700 1 -- 172.21.15.173:0/3047394701 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d0107360 con 0x7f14d00f6510 2024-02-19T21:56:44.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.688+0000 7f14ccff9700 1 -- 172.21.15.173:0/3047394701 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14c00095f0 con 0x7f14d00f6510 2024-02-19T21:56:44.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.689+0000 7f14d68ff700 1 -- 172.21.15.173:0/3047394701 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d0045ad0 con 0x7f14d00f6510 2024-02-19T21:56:44.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.689+0000 7f14ccff9700 1 --2- 172.21.15.173:0/3047394701 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14bc02ff10 0x7f14bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:44.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.689+0000 7f14ccff9700 1 -- 172.21.15.173:0/3047394701 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14c0040f80 con 0x7f14d00f6510 2024-02-19T21:56:44.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.689+0000 7f14cf7fe700 1 --2- 172.21.15.173:0/3047394701 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14bc02ff10 0x7f14bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:44.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.689+0000 7f14cf7fe700 1 --2- 172.21.15.173:0/3047394701 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14bc02ff10 0x7f14bc0323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f14b80096a0 tx=0x7f14b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:44.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.694+0000 7f14ccff9700 1 -- 172.21.15.173:0/3047394701 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14c001dbc0 con 0x7f14d00f6510 2024-02-19T21:56:44.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.897+0000 7f14d68ff700 1 -- 172.21.15.173:0/3047394701 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14d0108210 con 0x7f14d00f6510 2024-02-19T21:56:44.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.898+0000 7f14ccff9700 1 -- 172.21.15.173:0/3047394701 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f14c0010230 con 0x7f14d00f6510 2024-02-19T21:56:44.900 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:44.900 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:44.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.899+0000 7f14d68ff700 1 -- 172.21.15.173:0/3047394701 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14bc02ff10 msgr2=0x7f14bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:44.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.899+0000 7f14d68ff700 1 --2- 172.21.15.173:0/3047394701 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14bc02ff10 0x7f14bc0323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f14b80096a0 tx=0x7f14b8006b40).stop 2024-02-19T21:56:44.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.899+0000 7f14d68ff700 1 -- 172.21.15.173:0/3047394701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14d00f6510 msgr2=0x7f14d01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:44.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.899+0000 7f14d68ff700 1 --2- 172.21.15.173:0/3047394701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14d00f6510 0x7f14d01067e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f14c0005010 tx=0x7f14c000b040).stop 2024-02-19T21:56:44.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.899+0000 7f14d68ff700 1 -- 172.21.15.173:0/3047394701 shutdown_connections 2024-02-19T21:56:44.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.899+0000 7f14d68ff700 1 --2- 172.21.15.173:0/3047394701 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14bc02ff10 0x7f14bc0323d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:44.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.899+0000 7f14d68ff700 1 --2- 172.21.15.173:0/3047394701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14d00f6510 0x7f14d01067e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:44.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.899+0000 7f14d68ff700 1 -- 172.21.15.173:0/3047394701 >> 172.21.15.173:0/3047394701 conn(0x7f14d00f1c70 msgr2=0x7f14d00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:44.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.899+0000 7f14d68ff700 1 -- 172.21.15.173:0/3047394701 shutdown_connections 2024-02-19T21:56:44.902 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:44.899+0000 7f14d68ff700 1 -- 172.21.15.173:0/3047394701 wait complete. 2024-02-19T21:56:44.902 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:42.460403+0000 mgr.smithi143.bissbc (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:45.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:45 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:44.898926+0000 mon.smithi143 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.173:0/3047394701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:46.568 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:46.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:44.460669+0000 mgr.smithi143.bissbc (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.472+0000 7f9919fe3700 1 -- 172.21.15.173:0/1607812304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9914060280 msgr2=0x7f99140f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.472+0000 7f9919fe3700 1 --2- 172.21.15.173:0/1607812304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9914060280 0x7f99140f90f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f9908009fb0 tx=0x7f9908009b30).stop 2024-02-19T21:56:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.472+0000 7f9919fe3700 1 -- 172.21.15.173:0/1607812304 shutdown_connections 2024-02-19T21:56:48.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.472+0000 7f9919fe3700 1 --2- 172.21.15.173:0/1607812304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9914060280 0x7f99140f90f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.472+0000 7f9919fe3700 1 -- 172.21.15.173:0/1607812304 >> 172.21.15.173:0/1607812304 conn(0x7f99140f0b70 msgr2=0x7f99140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.472+0000 7f9919fe3700 1 -- 172.21.15.173:0/1607812304 shutdown_connections 2024-02-19T21:56:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.472+0000 7f9919fe3700 1 -- 172.21.15.173:0/1607812304 wait complete. 2024-02-19T21:56:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.473+0000 7f9919fe3700 1 Processor -- start 2024-02-19T21:56:48.475 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.473+0000 7f9919fe3700 1 -- start start 2024-02-19T21:56:48.475 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.473+0000 7f9919fe3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9914060280 0x7f9914106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:48.475 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.473+0000 7f9919fe3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9914106c70 con 0x7f9914060280 2024-02-19T21:56:48.475 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.473+0000 7f99137fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9914060280 0x7f9914106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:48.475 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.473+0000 7f99137fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9914060280 0x7f9914106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:54008/0 (socket says 172.21.15.173:54008) 2024-02-19T21:56:48.475 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.473+0000 7f99137fe700 1 -- 172.21.15.173:0/3707078077 learned_addr learned my addr 172.21.15.173:0/3707078077 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.473+0000 7f99137fe700 1 -- 172.21.15.173:0/3707078077 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9908005040 con 0x7f9914060280 2024-02-19T21:56:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.474+0000 7f99137fe700 1 --2- 172.21.15.173:0/3707078077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9914060280 0x7f9914106780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f99080089f0 tx=0x7f990800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.474+0000 7f98fbfff700 1 -- 172.21.15.173:0/3707078077 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9908009320 con 0x7f9914060280 2024-02-19T21:56:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.474+0000 7f98fbfff700 1 -- 172.21.15.173:0/3707078077 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9908013380 con 0x7f9914060280 2024-02-19T21:56:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.474+0000 7f9919fe3700 1 -- 172.21.15.173:0/3707078077 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9914106e70 con 0x7f9914060280 2024-02-19T21:56:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.474+0000 7f98fbfff700 1 -- 172.21.15.173:0/3707078077 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9908014e20 con 0x7f9914060280 2024-02-19T21:56:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.474+0000 7f9919fe3700 1 -- 172.21.15.173:0/3707078077 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9914107300 con 0x7f9914060280 2024-02-19T21:56:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.474+0000 7f98fbfff700 1 -- 172.21.15.173:0/3707078077 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9908014420 con 0x7f9914060280 2024-02-19T21:56:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.475+0000 7f9919fe3700 1 -- 172.21.15.173:0/3707078077 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9914045ad0 con 0x7f9914060280 2024-02-19T21:56:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.475+0000 7f98fbfff700 1 --2- 172.21.15.173:0/3707078077 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f98fc02ff10 0x7f98fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.475+0000 7f98fbfff700 1 -- 172.21.15.173:0/3707078077 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9908040ff0 con 0x7f9914060280 2024-02-19T21:56:48.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.475+0000 7f9912ffd700 1 --2- 172.21.15.173:0/3707078077 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f98fc02ff10 0x7f98fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:48.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.475+0000 7f9912ffd700 1 --2- 172.21.15.173:0/3707078077 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f98fc02ff10 0x7f98fc0323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f99040096a0 tx=0x7f9904006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:48.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.480+0000 7f98fbfff700 1 -- 172.21.15.173:0/3707078077 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99080095f0 con 0x7f9914060280 2024-02-19T21:56:48.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.685+0000 7f9919fe3700 1 -- 172.21.15.173:0/3707078077 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99141081b0 con 0x7f9914060280 2024-02-19T21:56:48.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.685+0000 7f98fbfff700 1 -- 172.21.15.173:0/3707078077 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f990801db70 con 0x7f9914060280 2024-02-19T21:56:48.685 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:48.685 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:48.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.687+0000 7f9919fe3700 1 -- 172.21.15.173:0/3707078077 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f98fc02ff10 msgr2=0x7f98fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:48.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.687+0000 7f9919fe3700 1 --2- 172.21.15.173:0/3707078077 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f98fc02ff10 0x7f98fc0323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f99040096a0 tx=0x7f9904006b40).stop 2024-02-19T21:56:48.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.687+0000 7f9919fe3700 1 -- 172.21.15.173:0/3707078077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9914060280 msgr2=0x7f9914106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:48.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.687+0000 7f9919fe3700 1 --2- 172.21.15.173:0/3707078077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9914060280 0x7f9914106780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f99080089f0 tx=0x7f990800b040).stop 2024-02-19T21:56:48.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.687+0000 7f9919fe3700 1 -- 172.21.15.173:0/3707078077 shutdown_connections 2024-02-19T21:56:48.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.687+0000 7f9919fe3700 1 --2- 172.21.15.173:0/3707078077 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f98fc02ff10 0x7f98fc0323d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:48.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.687+0000 7f9919fe3700 1 --2- 172.21.15.173:0/3707078077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9914060280 0x7f9914106780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:48.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.687+0000 7f9919fe3700 1 -- 172.21.15.173:0/3707078077 >> 172.21.15.173:0/3707078077 conn(0x7f99140f0b70 msgr2=0x7f99140f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:48.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.687+0000 7f9919fe3700 1 -- 172.21.15.173:0/3707078077 shutdown_connections 2024-02-19T21:56:48.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:48.687+0000 7f9919fe3700 1 -- 172.21.15.173:0/3707078077 wait complete. 2024-02-19T21:56:48.688 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:46.460941+0000 mgr.smithi143.bissbc (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:49.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:49 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:48.686490+0000 mon.smithi143 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.173:0/3707078077' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:50.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:50.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:48.461190+0000 mgr.smithi143.bissbc (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:51.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:51 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:50.461544+0000 mgr.smithi143.bissbc (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:51.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.997+0000 7f6ba8f94700 1 -- 172.21.15.173:0/82498364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40f6510 msgr2=0x7f6ba40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:51.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.997+0000 7f6ba8f94700 1 --2- 172.21.15.173:0/82498364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40f6510 0x7f6ba40f6930 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f6b94009fb0 tx=0x7f6b94009b30).stop 2024-02-19T21:56:51.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.997+0000 7f6ba8f94700 1 -- 172.21.15.173:0/82498364 shutdown_connections 2024-02-19T21:56:51.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.997+0000 7f6ba8f94700 1 --2- 172.21.15.173:0/82498364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40f6510 0x7f6ba40f6930 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:51.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.997+0000 7f6ba8f94700 1 -- 172.21.15.173:0/82498364 >> 172.21.15.173:0/82498364 conn(0x7f6ba40f1c70 msgr2=0x7f6ba40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:51.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.997+0000 7f6ba8f94700 1 -- 172.21.15.173:0/82498364 shutdown_connections 2024-02-19T21:56:51.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.997+0000 7f6ba8f94700 1 -- 172.21.15.173:0/82498364 wait complete. 2024-02-19T21:56:51.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.998+0000 7f6ba8f94700 1 Processor -- start 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.998+0000 7f6ba8f94700 1 -- start start 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.998+0000 7f6ba8f94700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40f6510 0x7f6ba41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.998+0000 7f6ba8f94700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba4106cd0 con 0x7f6ba40f6510 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.998+0000 7f6ba259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40f6510 0x7f6ba41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.998+0000 7f6ba259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40f6510 0x7f6ba41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43598/0 (socket says 172.21.15.173:43598) 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.998+0000 7f6ba259c700 1 -- 172.21.15.173:0/3434093913 learned_addr learned my addr 172.21.15.173:0/3434093913 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.999+0000 7f6ba259c700 1 -- 172.21.15.173:0/3434093913 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b94005040 con 0x7f6ba40f6510 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.999+0000 7f6ba259c700 1 --2- 172.21.15.173:0/3434093913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40f6510 0x7f6ba41067e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f6b94005010 tx=0x7f6b9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.999+0000 7f6b92ffd700 1 -- 172.21.15.173:0/3434093913 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b94009320 con 0x7f6ba40f6510 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.999+0000 7f6ba8f94700 1 -- 172.21.15.173:0/3434093913 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba4106ed0 con 0x7f6ba40f6510 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.999+0000 7f6b92ffd700 1 -- 172.21.15.173:0/3434093913 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b94013380 con 0x7f6ba40f6510 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.999+0000 7f6b92ffd700 1 -- 172.21.15.173:0/3434093913 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b94014e50 con 0x7f6ba40f6510 2024-02-19T21:56:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:51.999+0000 7f6ba8f94700 1 -- 172.21.15.173:0/3434093913 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba4107360 con 0x7f6ba40f6510 2024-02-19T21:56:52.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.000+0000 7f6b92ffd700 1 -- 172.21.15.173:0/3434093913 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b940095f0 con 0x7f6ba40f6510 2024-02-19T21:56:52.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.000+0000 7f6ba8f94700 1 -- 172.21.15.173:0/3434093913 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba4045ad0 con 0x7f6ba40f6510 2024-02-19T21:56:52.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.000+0000 7f6b92ffd700 1 --2- 172.21.15.173:0/3434093913 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6b8c02ff10 0x7f6b8c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:52.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.000+0000 7f6b92ffd700 1 -- 172.21.15.173:0/3434093913 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b94040e90 con 0x7f6ba40f6510 2024-02-19T21:56:52.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.000+0000 7f6ba1d9b700 1 --2- 172.21.15.173:0/3434093913 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6b8c02ff10 0x7f6b8c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:52.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.000+0000 7f6ba1d9b700 1 --2- 172.21.15.173:0/3434093913 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6b8c02ff10 0x7f6b8c0323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f6b980096a0 tx=0x7f6b98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:52.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.005+0000 7f6b92ffd700 1 -- 172.21.15.173:0/3434093913 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b94014420 con 0x7f6ba40f6510 2024-02-19T21:56:52.212 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.212+0000 7f6ba8f94700 1 -- 172.21.15.173:0/3434093913 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ba4108210 con 0x7f6ba40f6510 2024-02-19T21:56:52.213 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.213+0000 7f6b92ffd700 1 -- 172.21.15.173:0/3434093913 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6b94010230 con 0x7f6ba40f6510 2024-02-19T21:56:52.214 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:52.214 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:52.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.214+0000 7f6ba8f94700 1 -- 172.21.15.173:0/3434093913 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6b8c02ff10 msgr2=0x7f6b8c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:52.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.214+0000 7f6ba8f94700 1 --2- 172.21.15.173:0/3434093913 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6b8c02ff10 0x7f6b8c0323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f6b980096a0 tx=0x7f6b98006b40).stop 2024-02-19T21:56:52.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.214+0000 7f6ba8f94700 1 -- 172.21.15.173:0/3434093913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40f6510 msgr2=0x7f6ba41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:52.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.214+0000 7f6ba8f94700 1 --2- 172.21.15.173:0/3434093913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40f6510 0x7f6ba41067e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f6b94005010 tx=0x7f6b9400b040).stop 2024-02-19T21:56:52.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.214+0000 7f6ba8f94700 1 -- 172.21.15.173:0/3434093913 shutdown_connections 2024-02-19T21:56:52.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.214+0000 7f6ba8f94700 1 --2- 172.21.15.173:0/3434093913 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6b8c02ff10 0x7f6b8c0323d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:52.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.214+0000 7f6ba8f94700 1 --2- 172.21.15.173:0/3434093913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ba40f6510 0x7f6ba41067e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:52.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.214+0000 7f6ba8f94700 1 -- 172.21.15.173:0/3434093913 >> 172.21.15.173:0/3434093913 conn(0x7f6ba40f1c70 msgr2=0x7f6ba40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:52.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.214+0000 7f6ba8f94700 1 -- 172.21.15.173:0/3434093913 shutdown_connections 2024-02-19T21:56:52.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:52.214+0000 7f6ba8f94700 1 -- 172.21.15.173:0/3434093913 wait complete. 2024-02-19T21:56:52.215 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:52.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:52.213821+0000 mon.smithi143 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.173:0/3434093913' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:53.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:53.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:52.461804+0000 mgr.smithi143.bissbc (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:55.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.962+0000 7f49bf77f700 1 -- 172.21.15.173:0/1661205820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b8060200 msgr2=0x7f49b80f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:55.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.962+0000 7f49bf77f700 1 --2- 172.21.15.173:0/1661205820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b8060200 0x7f49b80f9170 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f49a8009fb0 tx=0x7f49a8009b30).stop 2024-02-19T21:56:55.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.962+0000 7f49bf77f700 1 -- 172.21.15.173:0/1661205820 shutdown_connections 2024-02-19T21:56:55.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.962+0000 7f49bf77f700 1 --2- 172.21.15.173:0/1661205820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b8060200 0x7f49b80f9170 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:55.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.962+0000 7f49bf77f700 1 -- 172.21.15.173:0/1661205820 >> 172.21.15.173:0/1661205820 conn(0x7f49b80f0b70 msgr2=0x7f49b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:55.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.962+0000 7f49bf77f700 1 -- 172.21.15.173:0/1661205820 shutdown_connections 2024-02-19T21:56:55.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.962+0000 7f49bf77f700 1 -- 172.21.15.173:0/1661205820 wait complete. 2024-02-19T21:56:55.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.963+0000 7f49bf77f700 1 Processor -- start 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.963+0000 7f49bf77f700 1 -- start start 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.963+0000 7f49bf77f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b8060200 0x7f49b80f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.963+0000 7f49bf77f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b80f5070 con 0x7f49b8060200 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.963+0000 7f49bd51b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b8060200 0x7f49b80f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.963+0000 7f49bd51b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b8060200 0x7f49b80f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43618/0 (socket says 172.21.15.173:43618) 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.963+0000 7f49bd51b700 1 -- 172.21.15.173:0/1211453768 learned_addr learned my addr 172.21.15.173:0/1211453768 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.963+0000 7f49bd51b700 1 -- 172.21.15.173:0/1211453768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a8005040 con 0x7f49b8060200 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.964+0000 7f49bd51b700 1 --2- 172.21.15.173:0/1211453768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b8060200 0x7f49b80f6bf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f49a80089f0 tx=0x7f49a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.964+0000 7f49adffb700 1 -- 172.21.15.173:0/1211453768 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49a8009320 con 0x7f49b8060200 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.964+0000 7f49bf77f700 1 -- 172.21.15.173:0/1211453768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b80f5210 con 0x7f49b8060200 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.964+0000 7f49adffb700 1 -- 172.21.15.173:0/1211453768 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49a8013380 con 0x7f49b8060200 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.964+0000 7f49adffb700 1 -- 172.21.15.173:0/1211453768 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49a8014e20 con 0x7f49b8060200 2024-02-19T21:56:55.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.964+0000 7f49bf77f700 1 -- 172.21.15.173:0/1211453768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b80f56a0 con 0x7f49b8060200 2024-02-19T21:56:55.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.965+0000 7f49adffb700 1 -- 172.21.15.173:0/1211453768 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49a80263a0 con 0x7f49b8060200 2024-02-19T21:56:55.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.965+0000 7f49bf77f700 1 -- 172.21.15.173:0/1211453768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b8059100 con 0x7f49b8060200 2024-02-19T21:56:55.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.965+0000 7f49adffb700 1 --2- 172.21.15.173:0/1211453768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f49a402ff10 0x7f49a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:55.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.965+0000 7f49adffb700 1 -- 172.21.15.173:0/1211453768 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49a8040f00 con 0x7f49b8060200 2024-02-19T21:56:55.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.965+0000 7f49bcd1a700 1 --2- 172.21.15.173:0/1211453768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f49a402ff10 0x7f49a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:55.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.966+0000 7f49bcd1a700 1 --2- 172.21.15.173:0/1211453768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f49a402ff10 0x7f49a40323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f49b40096a0 tx=0x7f49b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:55.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:55.970+0000 7f49adffb700 1 -- 172.21.15.173:0/1211453768 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49a80126f0 con 0x7f49b8060200 2024-02-19T21:56:56.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.178+0000 7f49bf77f700 1 -- 172.21.15.173:0/1211453768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49b80fc090 con 0x7f49b8060200 2024-02-19T21:56:56.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.178+0000 7f49adffb700 1 -- 172.21.15.173:0/1211453768 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f49a80126f0 con 0x7f49b8060200 2024-02-19T21:56:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:56.178 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:56.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.180+0000 7f49bf77f700 1 -- 172.21.15.173:0/1211453768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f49a402ff10 msgr2=0x7f49a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:56.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.180+0000 7f49bf77f700 1 --2- 172.21.15.173:0/1211453768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f49a402ff10 0x7f49a40323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f49b40096a0 tx=0x7f49b4006b40).stop 2024-02-19T21:56:56.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.180+0000 7f49bf77f700 1 -- 172.21.15.173:0/1211453768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b8060200 msgr2=0x7f49b80f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:56.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.180+0000 7f49bf77f700 1 --2- 172.21.15.173:0/1211453768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b8060200 0x7f49b80f6bf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f49a80089f0 tx=0x7f49a800b040).stop 2024-02-19T21:56:56.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.180+0000 7f49bf77f700 1 -- 172.21.15.173:0/1211453768 shutdown_connections 2024-02-19T21:56:56.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.180+0000 7f49bf77f700 1 --2- 172.21.15.173:0/1211453768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f49a402ff10 0x7f49a40323d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:56.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.180+0000 7f49bf77f700 1 --2- 172.21.15.173:0/1211453768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b8060200 0x7f49b80f6bf0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:56.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.180+0000 7f49bf77f700 1 -- 172.21.15.173:0/1211453768 >> 172.21.15.173:0/1211453768 conn(0x7f49b80f0b70 msgr2=0x7f49b80f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:56.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.180+0000 7f49bf77f700 1 -- 172.21.15.173:0/1211453768 shutdown_connections 2024-02-19T21:56:56.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:56.180+0000 7f49bf77f700 1 -- 172.21.15.173:0/1211453768 wait complete. 2024-02-19T21:56:56.181 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:56:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:54.461999+0000 mgr.smithi143.bissbc (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:56.179668+0000 mon.smithi143 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.173:0/1211453768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:56:57.857 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:56:57.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:56:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:56:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:56.462298+0000 mgr.smithi143.bissbc (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:59.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.741+0000 7f6225112700 1 -- 172.21.15.173:0/3599362836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62200f62a0 msgr2=0x7f62200f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:59.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.741+0000 7f6225112700 1 --2- 172.21.15.173:0/3599362836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62200f62a0 0x7f62200f66c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f6210009fb0 tx=0x7f6210009b30).stop 2024-02-19T21:56:59.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.741+0000 7f6225112700 1 -- 172.21.15.173:0/3599362836 shutdown_connections 2024-02-19T21:56:59.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.741+0000 7f6225112700 1 --2- 172.21.15.173:0/3599362836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62200f62a0 0x7f62200f66c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:59.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.741+0000 7f6225112700 1 -- 172.21.15.173:0/3599362836 >> 172.21.15.173:0/3599362836 conn(0x7f62200f1aa0 msgr2=0x7f62200f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:59.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.741+0000 7f6225112700 1 -- 172.21.15.173:0/3599362836 shutdown_connections 2024-02-19T21:56:59.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.741+0000 7f6225112700 1 -- 172.21.15.173:0/3599362836 wait complete. 2024-02-19T21:56:59.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.742+0000 7f6225112700 1 Processor -- start 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.742+0000 7f6225112700 1 -- start start 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.742+0000 7f6225112700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62200f62a0 0x7f622006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.742+0000 7f6225112700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6220069e20 con 0x7f62200f62a0 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.742+0000 7f621ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62200f62a0 0x7f622006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.742+0000 7f621ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62200f62a0 0x7f622006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43634/0 (socket says 172.21.15.173:43634) 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.742+0000 7f621ffff700 1 -- 172.21.15.173:0/119131467 learned_addr learned my addr 172.21.15.173:0/119131467 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.743+0000 7f621ffff700 1 -- 172.21.15.173:0/119131467 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6210005040 con 0x7f62200f62a0 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.743+0000 7f621ffff700 1 --2- 172.21.15.173:0/119131467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62200f62a0 0x7f622006b9a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f62100089f0 tx=0x7f621000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.743+0000 7f621cff9700 1 -- 172.21.15.173:0/119131467 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6210009320 con 0x7f62200f62a0 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.743+0000 7f6225112700 1 -- 172.21.15.173:0/119131467 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6220069fc0 con 0x7f62200f62a0 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.743+0000 7f621cff9700 1 -- 172.21.15.173:0/119131467 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6210013380 con 0x7f62200f62a0 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.743+0000 7f621cff9700 1 -- 172.21.15.173:0/119131467 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6210014e20 con 0x7f62200f62a0 2024-02-19T21:56:59.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.743+0000 7f6225112700 1 -- 172.21.15.173:0/119131467 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622006a450 con 0x7f62200f62a0 2024-02-19T21:56:59.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.744+0000 7f621cff9700 1 -- 172.21.15.173:0/119131467 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6210014420 con 0x7f62200f62a0 2024-02-19T21:56:59.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.744+0000 7f621cff9700 1 --2- 172.21.15.173:0/119131467 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f620802ff10 0x7f62080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:59.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.744+0000 7f621cff9700 1 -- 172.21.15.173:0/119131467 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6210040f50 con 0x7f62200f62a0 2024-02-19T21:56:59.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.744+0000 7f6225112700 1 -- 172.21.15.173:0/119131467 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6220059100 con 0x7f62200f62a0 2024-02-19T21:56:59.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.744+0000 7f621f7fe700 1 --2- 172.21.15.173:0/119131467 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f620802ff10 0x7f62080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:59.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.744+0000 7f621f7fe700 1 --2- 172.21.15.173:0/119131467 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f620802ff10 0x7f62080323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f62140096a0 tx=0x7f6214006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:59.750 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.750+0000 7f621cff9700 1 -- 172.21.15.173:0/119131467 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62100095f0 con 0x7f62200f62a0 2024-02-19T21:56:59.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.957+0000 7f6225112700 1 -- 172.21.15.173:0/119131467 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f622006ab50 con 0x7f62200f62a0 2024-02-19T21:56:59.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.957+0000 7f621cff9700 1 -- 172.21.15.173:0/119131467 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f621001db70 con 0x7f62200f62a0 2024-02-19T21:56:59.958 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:56:59.958 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:56:59.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.959+0000 7f6225112700 1 -- 172.21.15.173:0/119131467 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f620802ff10 msgr2=0x7f62080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:59.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.959+0000 7f6225112700 1 --2- 172.21.15.173:0/119131467 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f620802ff10 0x7f62080323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f62140096a0 tx=0x7f6214006b40).stop 2024-02-19T21:56:59.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.959+0000 7f6225112700 1 -- 172.21.15.173:0/119131467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62200f62a0 msgr2=0x7f622006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:59.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.959+0000 7f6225112700 1 --2- 172.21.15.173:0/119131467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62200f62a0 0x7f622006b9a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f62100089f0 tx=0x7f621000b040).stop 2024-02-19T21:56:59.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.959+0000 7f6225112700 1 -- 172.21.15.173:0/119131467 shutdown_connections 2024-02-19T21:56:59.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.959+0000 7f6225112700 1 --2- 172.21.15.173:0/119131467 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f620802ff10 0x7f62080323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:59.960 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.959+0000 7f6225112700 1 --2- 172.21.15.173:0/119131467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62200f62a0 0x7f622006b9a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:59.961 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.959+0000 7f6225112700 1 -- 172.21.15.173:0/119131467 >> 172.21.15.173:0/119131467 conn(0x7f62200f1aa0 msgr2=0x7f62200f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:59.961 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.959+0000 7f6225112700 1 -- 172.21.15.173:0/119131467 shutdown_connections 2024-02-19T21:56:59.961 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:56:59.959+0000 7f6225112700 1 -- 172.21.15.173:0/119131467 wait complete. 2024-02-19T21:56:59.961 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:56:58.462595+0000 mgr.smithi143.bissbc (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:56:59.959518+0000 mon.smithi143 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.173:0/119131467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:01.654 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:01.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:00.463008+0000 mgr.smithi143.bissbc (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:03.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.570+0000 7fe29b59e700 1 -- 172.21.15.173:0/3729779466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe29c0f51a0 msgr2=0x7fe29c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:03.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.570+0000 7fe29b59e700 1 --2- 172.21.15.173:0/3729779466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe29c0f51a0 0x7fe29c0f55c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fe28c009fb0 tx=0x7fe28c009b30).stop 2024-02-19T21:57:03.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.570+0000 7fe29b59e700 1 -- 172.21.15.173:0/3729779466 shutdown_connections 2024-02-19T21:57:03.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.570+0000 7fe29b59e700 1 --2- 172.21.15.173:0/3729779466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe29c0f51a0 0x7fe29c0f55c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:03.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.570+0000 7fe29b59e700 1 -- 172.21.15.173:0/3729779466 >> 172.21.15.173:0/3729779466 conn(0x7fe29c0f09a0 msgr2=0x7fe29c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:03.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.570+0000 7fe29b59e700 1 -- 172.21.15.173:0/3729779466 shutdown_connections 2024-02-19T21:57:03.570 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.570+0000 7fe29b59e700 1 -- 172.21.15.173:0/3729779466 wait complete. 2024-02-19T21:57:03.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.571+0000 7fe29b59e700 1 Processor -- start 2024-02-19T21:57:03.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.571+0000 7fe29b59e700 1 -- start start 2024-02-19T21:57:03.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.571+0000 7fe29b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe29c0f51a0 0x7fe29c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.571+0000 7fe29b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c1059c0 con 0x7fe29c0f51a0 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.571+0000 7fe29a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe29c0f51a0 0x7fe29c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.571+0000 7fe29a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe29c0f51a0 0x7fe29c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:58232/0 (socket says 172.21.15.173:58232) 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.571+0000 7fe29a59c700 1 -- 172.21.15.173:0/1694942584 learned_addr learned my addr 172.21.15.173:0/1694942584 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.571+0000 7fe29a59c700 1 -- 172.21.15.173:0/1694942584 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c005040 con 0x7fe29c0f51a0 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.572+0000 7fe29a59c700 1 --2- 172.21.15.173:0/1694942584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe29c0f51a0 0x7fe29c1054d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fe28c0089f0 tx=0x7fe28c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.572+0000 7fe28affd700 1 -- 172.21.15.173:0/1694942584 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe28c009320 con 0x7fe29c0f51a0 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.572+0000 7fe28affd700 1 -- 172.21.15.173:0/1694942584 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe28c013380 con 0x7fe29c0f51a0 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.572+0000 7fe29b59e700 1 -- 172.21.15.173:0/1694942584 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29c105bc0 con 0x7fe29c0f51a0 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.572+0000 7fe28affd700 1 -- 172.21.15.173:0/1694942584 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe28c014e20 con 0x7fe29c0f51a0 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.572+0000 7fe29b59e700 1 -- 172.21.15.173:0/1694942584 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c106050 con 0x7fe29c0f51a0 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.572+0000 7fe28affd700 1 -- 172.21.15.173:0/1694942584 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe28c0095f0 con 0x7fe29c0f51a0 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.573+0000 7fe28affd700 1 --2- 172.21.15.173:0/1694942584 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe28402ff60 0x7fe284032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:03.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.573+0000 7fe28affd700 1 -- 172.21.15.173:0/1694942584 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe28c040e40 con 0x7fe29c0f51a0 2024-02-19T21:57:03.574 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.573+0000 7fe29b59e700 1 -- 172.21.15.173:0/1694942584 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29c045ad0 con 0x7fe29c0f51a0 2024-02-19T21:57:03.574 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.573+0000 7fe299d9b700 1 --2- 172.21.15.173:0/1694942584 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe28402ff60 0x7fe284032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:03.574 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.573+0000 7fe299d9b700 1 --2- 172.21.15.173:0/1694942584 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe28402ff60 0x7fe284032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fe2900096a0 tx=0x7fe290006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:03.577 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.578+0000 7fe28affd700 1 -- 172.21.15.173:0/1694942584 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe28c014420 con 0x7fe29c0f51a0 2024-02-19T21:57:03.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.785+0000 7fe29b59e700 1 -- 172.21.15.173:0/1694942584 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe29c106d70 con 0x7fe29c0f51a0 2024-02-19T21:57:03.785 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.785+0000 7fe28affd700 1 -- 172.21.15.173:0/1694942584 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe28c02b020 con 0x7fe29c0f51a0 2024-02-19T21:57:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:03.785 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:03.787 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.787+0000 7fe29b59e700 1 -- 172.21.15.173:0/1694942584 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe28402ff60 msgr2=0x7fe284032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:03.787 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.787+0000 7fe29b59e700 1 --2- 172.21.15.173:0/1694942584 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe28402ff60 0x7fe284032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fe2900096a0 tx=0x7fe290006b40).stop 2024-02-19T21:57:03.787 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.787+0000 7fe29b59e700 1 -- 172.21.15.173:0/1694942584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe29c0f51a0 msgr2=0x7fe29c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:03.787 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.787+0000 7fe29b59e700 1 --2- 172.21.15.173:0/1694942584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe29c0f51a0 0x7fe29c1054d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fe28c0089f0 tx=0x7fe28c00b040).stop 2024-02-19T21:57:03.787 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.787+0000 7fe29b59e700 1 -- 172.21.15.173:0/1694942584 shutdown_connections 2024-02-19T21:57:03.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.787+0000 7fe29b59e700 1 --2- 172.21.15.173:0/1694942584 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe28402ff60 0x7fe284032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:03.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.787+0000 7fe29b59e700 1 --2- 172.21.15.173:0/1694942584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe29c0f51a0 0x7fe29c1054d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:03.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.787+0000 7fe29b59e700 1 -- 172.21.15.173:0/1694942584 >> 172.21.15.173:0/1694942584 conn(0x7fe29c0f09a0 msgr2=0x7fe29c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:03.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.787+0000 7fe29b59e700 1 -- 172.21.15.173:0/1694942584 shutdown_connections 2024-02-19T21:57:03.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:03.787+0000 7fe29b59e700 1 -- 172.21.15.173:0/1694942584 wait complete. 2024-02-19T21:57:03.788 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:02.463232+0000 mgr.smithi143.bissbc (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:04.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:03.786426+0000 mon.smithi143 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.173:0/1694942584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:05.521 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:05.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:04.463462+0000 mgr.smithi143.bissbc (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:07.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.607+0000 7f78abb2a700 1 -- 172.21.15.173:0/3032416300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a40f5410 msgr2=0x7f78a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:07.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.607+0000 7f78abb2a700 1 --2- 172.21.15.173:0/3032416300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a40f5410 0x7f78a40f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f789c009960 tx=0x7f789c004d10).stop 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.607+0000 7f78abb2a700 1 -- 172.21.15.173:0/3032416300 shutdown_connections 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.607+0000 7f78abb2a700 1 --2- 172.21.15.173:0/3032416300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a40f5410 0x7f78a40f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.607+0000 7f78abb2a700 1 -- 172.21.15.173:0/3032416300 >> 172.21.15.173:0/3032416300 conn(0x7f78a40f0b70 msgr2=0x7f78a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.607+0000 7f78abb2a700 1 -- 172.21.15.173:0/3032416300 shutdown_connections 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.607+0000 7f78abb2a700 1 -- 172.21.15.173:0/3032416300 wait complete. 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.607+0000 7f78abb2a700 1 Processor -- start 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.607+0000 7f78abb2a700 1 -- start start 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f78abb2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a40f5410 0x7f78a41094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f78abb2a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a4060240 con 0x7f78a40f5410 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f78a98c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a40f5410 0x7f78a41094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f78a98c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a40f5410 0x7f78a41094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:58248/0 (socket says 172.21.15.173:58248) 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f78a98c6700 1 -- 172.21.15.173:0/2317804870 learned_addr learned my addr 172.21.15.173:0/2317804870 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:07.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f78a98c6700 1 -- 172.21.15.173:0/2317804870 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c009640 con 0x7f78a40f5410 2024-02-19T21:57:07.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f78a98c6700 1 --2- 172.21.15.173:0/2317804870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a40f5410 0x7f78a41094e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f789c004b70 tx=0x7f789c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:07.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f789a7fc700 1 -- 172.21.15.173:0/2317804870 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f789c018070 con 0x7f78a40f5410 2024-02-19T21:57:07.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f78abb2a700 1 -- 172.21.15.173:0/2317804870 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a4060440 con 0x7f78a40f5410 2024-02-19T21:57:07.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f789a7fc700 1 -- 172.21.15.173:0/2317804870 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f789c00b970 con 0x7f78a40f5410 2024-02-19T21:57:07.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f789a7fc700 1 -- 172.21.15.173:0/2317804870 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f789c01c560 con 0x7f78a40f5410 2024-02-19T21:57:07.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.608+0000 7f78abb2a700 1 -- 172.21.15.173:0/2317804870 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a4109b90 con 0x7f78a40f5410 2024-02-19T21:57:07.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.609+0000 7f789a7fc700 1 -- 172.21.15.173:0/2317804870 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f789c013630 con 0x7f78a40f5410 2024-02-19T21:57:07.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.609+0000 7f789a7fc700 1 --2- 172.21.15.173:0/2317804870 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f789002ff60 0x7f7890032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:07.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.609+0000 7f78abb2a700 1 -- 172.21.15.173:0/2317804870 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a4045ad0 con 0x7f78a40f5410 2024-02-19T21:57:07.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.609+0000 7f789a7fc700 1 -- 172.21.15.173:0/2317804870 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f789c041320 con 0x7f78a40f5410 2024-02-19T21:57:07.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.610+0000 7f78a90c5700 1 --2- 172.21.15.173:0/2317804870 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f789002ff60 0x7f7890032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:07.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.610+0000 7f78a90c5700 1 --2- 172.21.15.173:0/2317804870 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f789002ff60 0x7f7890032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f78940096a0 tx=0x7f7894006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:07.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.614+0000 7f789a7fc700 1 -- 172.21.15.173:0/2317804870 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f789c012970 con 0x7f78a40f5410 2024-02-19T21:57:07.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.821+0000 7f78abb2a700 1 -- 172.21.15.173:0/2317804870 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78a410a3b0 con 0x7f78a40f5410 2024-02-19T21:57:07.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.822+0000 7f789a7fc700 1 -- 172.21.15.173:0/2317804870 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f789c012380 con 0x7f78a40f5410 2024-02-19T21:57:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:07.822 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:07.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.824+0000 7f78abb2a700 1 -- 172.21.15.173:0/2317804870 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f789002ff60 msgr2=0x7f7890032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:07.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.824+0000 7f78abb2a700 1 --2- 172.21.15.173:0/2317804870 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f789002ff60 0x7f7890032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f78940096a0 tx=0x7f7894006b40).stop 2024-02-19T21:57:07.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.824+0000 7f78abb2a700 1 -- 172.21.15.173:0/2317804870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a40f5410 msgr2=0x7f78a41094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:07.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.824+0000 7f78abb2a700 1 --2- 172.21.15.173:0/2317804870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a40f5410 0x7f78a41094e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f789c004b70 tx=0x7f789c004d10).stop 2024-02-19T21:57:07.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.824+0000 7f78abb2a700 1 -- 172.21.15.173:0/2317804870 shutdown_connections 2024-02-19T21:57:07.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.824+0000 7f78abb2a700 1 --2- 172.21.15.173:0/2317804870 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f789002ff60 0x7f7890032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:07.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.824+0000 7f78abb2a700 1 --2- 172.21.15.173:0/2317804870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a40f5410 0x7f78a41094e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:07.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.824+0000 7f78abb2a700 1 -- 172.21.15.173:0/2317804870 >> 172.21.15.173:0/2317804870 conn(0x7f78a40f0b70 msgr2=0x7f78a40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:07.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.824+0000 7f78abb2a700 1 -- 172.21.15.173:0/2317804870 shutdown_connections 2024-02-19T21:57:07.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:07.824+0000 7f78abb2a700 1 -- 172.21.15.173:0/2317804870 wait complete. 2024-02-19T21:57:07.825 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:06.463859+0000 mgr.smithi143.bissbc (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:07.823136+0000 mon.smithi143 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.173:0/2317804870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:09.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:09.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:08.464085+0000 mgr.smithi143.bissbc (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.334+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/1246592319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6f40f5410 msgr2=0x7fc6f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.334+0000 7fc6fb1cd700 1 --2- 172.21.15.173:0/1246592319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6f40f5410 0x7fc6f40f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fc6e8009fb0 tx=0x7fc6e8009b30).stop 2024-02-19T21:57:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.334+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/1246592319 shutdown_connections 2024-02-19T21:57:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.334+0000 7fc6fb1cd700 1 --2- 172.21.15.173:0/1246592319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6f40f5410 0x7fc6f40f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.334+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/1246592319 >> 172.21.15.173:0/1246592319 conn(0x7fc6f40f0b70 msgr2=0x7fc6f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.334+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/1246592319 shutdown_connections 2024-02-19T21:57:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.335+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/1246592319 wait complete. 2024-02-19T21:57:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.335+0000 7fc6fb1cd700 1 Processor -- start 2024-02-19T21:57:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.335+0000 7fc6fb1cd700 1 -- start start 2024-02-19T21:57:11.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.335+0000 7fc6fb1cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6f40f5410 0x7fc6f41067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.335+0000 7fc6fb1cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f4106ca0 con 0x7fc6f40f5410 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.336+0000 7fc6f8f69700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6f40f5410 0x7fc6f41067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.336+0000 7fc6f8f69700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6f40f5410 0x7fc6f41067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:45278/0 (socket says 172.21.15.173:45278) 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.336+0000 7fc6f8f69700 1 -- 172.21.15.173:0/618035969 learned_addr learned my addr 172.21.15.173:0/618035969 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.336+0000 7fc6f8f69700 1 -- 172.21.15.173:0/618035969 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e8005040 con 0x7fc6f40f5410 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.336+0000 7fc6f8f69700 1 --2- 172.21.15.173:0/618035969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6f40f5410 0x7fc6f41067b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fc6e8005010 tx=0x7fc6e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.336+0000 7fc6f17fa700 1 -- 172.21.15.173:0/618035969 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6e8009320 con 0x7fc6f40f5410 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.336+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/618035969 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f4106ea0 con 0x7fc6f40f5410 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.336+0000 7fc6f17fa700 1 -- 172.21.15.173:0/618035969 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6e8013380 con 0x7fc6f40f5410 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.336+0000 7fc6f17fa700 1 -- 172.21.15.173:0/618035969 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6e8014e50 con 0x7fc6f40f5410 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.336+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/618035969 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f4107330 con 0x7fc6f40f5410 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.337+0000 7fc6f17fa700 1 -- 172.21.15.173:0/618035969 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6e80263a0 con 0x7fc6f40f5410 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.337+0000 7fc6f17fa700 1 --2- 172.21.15.173:0/618035969 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc6e402ff10 0x7fc6e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:11.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.337+0000 7fc6f17fa700 1 -- 172.21.15.173:0/618035969 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6e8040e90 con 0x7fc6f40f5410 2024-02-19T21:57:11.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.337+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/618035969 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f4045ad0 con 0x7fc6f40f5410 2024-02-19T21:57:11.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.337+0000 7fc6f3fff700 1 --2- 172.21.15.173:0/618035969 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc6e402ff10 0x7fc6e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:11.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.338+0000 7fc6f3fff700 1 --2- 172.21.15.173:0/618035969 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc6e402ff10 0x7fc6e40323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc6e00096a0 tx=0x7fc6e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:11.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.342+0000 7fc6f17fa700 1 -- 172.21.15.173:0/618035969 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6e8012920 con 0x7fc6f40f5410 2024-02-19T21:57:11.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.551+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/618035969 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6f4108050 con 0x7fc6f40f5410 2024-02-19T21:57:11.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.552+0000 7fc6f17fa700 1 -- 172.21.15.173:0/618035969 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc6e8012330 con 0x7fc6f40f5410 2024-02-19T21:57:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:11.552 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:11.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.553+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/618035969 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc6e402ff10 msgr2=0x7fc6e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:11.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.553+0000 7fc6fb1cd700 1 --2- 172.21.15.173:0/618035969 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc6e402ff10 0x7fc6e40323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc6e00096a0 tx=0x7fc6e0006b40).stop 2024-02-19T21:57:11.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.553+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/618035969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6f40f5410 msgr2=0x7fc6f41067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:11.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.553+0000 7fc6fb1cd700 1 --2- 172.21.15.173:0/618035969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6f40f5410 0x7fc6f41067b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fc6e8005010 tx=0x7fc6e800b040).stop 2024-02-19T21:57:11.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.554+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/618035969 shutdown_connections 2024-02-19T21:57:11.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.554+0000 7fc6fb1cd700 1 --2- 172.21.15.173:0/618035969 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc6e402ff10 0x7fc6e40323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:11.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.554+0000 7fc6fb1cd700 1 --2- 172.21.15.173:0/618035969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6f40f5410 0x7fc6f41067b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:11.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.554+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/618035969 >> 172.21.15.173:0/618035969 conn(0x7fc6f40f0b70 msgr2=0x7fc6f405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:11.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.554+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/618035969 shutdown_connections 2024-02-19T21:57:11.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:11.554+0000 7fc6fb1cd700 1 -- 172.21.15.173:0/618035969 wait complete. 2024-02-19T21:57:11.557 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:12.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:10.464350+0000 mgr.smithi143.bissbc (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:12.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:11.553013+0000 mon.smithi143 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.173:0/618035969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:13.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:13.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:12.464646+0000 mgr.smithi143.bissbc (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:15.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.099+0000 7f455458a700 1 -- 172.21.15.173:0/2634595733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f454c0f6510 msgr2=0x7f454c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:15.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.099+0000 7f455458a700 1 --2- 172.21.15.173:0/2634595733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f454c0f6510 0x7f454c0f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4540009fb0 tx=0x7f4540009b30).stop 2024-02-19T21:57:15.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.099+0000 7f455458a700 1 -- 172.21.15.173:0/2634595733 shutdown_connections 2024-02-19T21:57:15.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.099+0000 7f455458a700 1 --2- 172.21.15.173:0/2634595733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f454c0f6510 0x7f454c0f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:15.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.099+0000 7f455458a700 1 -- 172.21.15.173:0/2634595733 >> 172.21.15.173:0/2634595733 conn(0x7f454c0f1c70 msgr2=0x7f454c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:15.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.099+0000 7f455458a700 1 -- 172.21.15.173:0/2634595733 shutdown_connections 2024-02-19T21:57:15.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.099+0000 7f455458a700 1 -- 172.21.15.173:0/2634595733 wait complete. 2024-02-19T21:57:15.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.100+0000 7f455458a700 1 Processor -- start 2024-02-19T21:57:15.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.100+0000 7f455458a700 1 -- start start 2024-02-19T21:57:15.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.100+0000 7f455458a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f454c0f6510 0x7f454c108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:15.101 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.100+0000 7f455458a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c108ef0 con 0x7f454c0f6510 2024-02-19T21:57:15.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.100+0000 7f4552326700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f454c0f6510 0x7f454c108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:15.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.100+0000 7f4552326700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f454c0f6510 0x7f454c108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:45302/0 (socket says 172.21.15.173:45302) 2024-02-19T21:57:15.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.100+0000 7f4552326700 1 -- 172.21.15.173:0/368713434 learned_addr learned my addr 172.21.15.173:0/368713434 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:15.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.100+0000 7f4552326700 1 -- 172.21.15.173:0/368713434 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4540005040 con 0x7f454c0f6510 2024-02-19T21:57:15.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.100+0000 7f4552326700 1 --2- 172.21.15.173:0/368713434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f454c0f6510 0x7f454c108a00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f45400089f0 tx=0x7f454000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:15.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.101+0000 7f453effd700 1 -- 172.21.15.173:0/368713434 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4540009320 con 0x7f454c0f6510 2024-02-19T21:57:15.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.101+0000 7f453effd700 1 -- 172.21.15.173:0/368713434 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4540013380 con 0x7f454c0f6510 2024-02-19T21:57:15.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.101+0000 7f455458a700 1 -- 172.21.15.173:0/368713434 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454c1090f0 con 0x7f454c0f6510 2024-02-19T21:57:15.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.101+0000 7f453effd700 1 -- 172.21.15.173:0/368713434 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f454001ed80 con 0x7f454c0f6510 2024-02-19T21:57:15.102 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.101+0000 7f455458a700 1 -- 172.21.15.173:0/368713434 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454c109580 con 0x7f454c0f6510 2024-02-19T21:57:15.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.101+0000 7f453effd700 1 -- 172.21.15.173:0/368713434 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f454001e420 con 0x7f454c0f6510 2024-02-19T21:57:15.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.101+0000 7f455458a700 1 -- 172.21.15.173:0/368713434 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f454c045ad0 con 0x7f454c0f6510 2024-02-19T21:57:15.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.102+0000 7f453effd700 1 --2- 172.21.15.173:0/368713434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f453802ff60 0x7f4538032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:15.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.102+0000 7f453effd700 1 -- 172.21.15.173:0/368713434 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4540040fc0 con 0x7f454c0f6510 2024-02-19T21:57:15.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.102+0000 7f4551b25700 1 --2- 172.21.15.173:0/368713434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f453802ff60 0x7f4538032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:15.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.102+0000 7f4551b25700 1 --2- 172.21.15.173:0/368713434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f453802ff60 0x7f4538032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f45480096a0 tx=0x7f4548006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:15.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.106+0000 7f453effd700 1 -- 172.21.15.173:0/368713434 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f454006d2b0 con 0x7f454c0f6510 2024-02-19T21:57:15.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.311+0000 7f455458a700 1 -- 172.21.15.173:0/368713434 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f454c10a430 con 0x7f454c0f6510 2024-02-19T21:57:15.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.312+0000 7f453effd700 1 -- 172.21.15.173:0/368713434 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f45400095f0 con 0x7f454c0f6510 2024-02-19T21:57:15.312 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:15.312 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:15.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.313+0000 7f455458a700 1 -- 172.21.15.173:0/368713434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f453802ff60 msgr2=0x7f4538032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:15.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.314+0000 7f455458a700 1 --2- 172.21.15.173:0/368713434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f453802ff60 0x7f4538032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f45480096a0 tx=0x7f4548006b40).stop 2024-02-19T21:57:15.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.314+0000 7f455458a700 1 -- 172.21.15.173:0/368713434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f454c0f6510 msgr2=0x7f454c108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:15.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.314+0000 7f455458a700 1 --2- 172.21.15.173:0/368713434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f454c0f6510 0x7f454c108a00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f45400089f0 tx=0x7f454000b040).stop 2024-02-19T21:57:15.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.314+0000 7f455458a700 1 -- 172.21.15.173:0/368713434 shutdown_connections 2024-02-19T21:57:15.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.314+0000 7f455458a700 1 --2- 172.21.15.173:0/368713434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f453802ff60 0x7f4538032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:15.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.314+0000 7f455458a700 1 --2- 172.21.15.173:0/368713434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f454c0f6510 0x7f454c108a00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:15.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.314+0000 7f455458a700 1 -- 172.21.15.173:0/368713434 >> 172.21.15.173:0/368713434 conn(0x7f454c0f1c70 msgr2=0x7f454c0f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:15.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.314+0000 7f455458a700 1 -- 172.21.15.173:0/368713434 shutdown_connections 2024-02-19T21:57:15.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:15.314+0000 7f455458a700 1 -- 172.21.15.173:0/368713434 wait complete. 2024-02-19T21:57:15.315 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:15.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:15 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:15.312893+0000 mon.smithi143 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.173:0/368713434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:14.464883+0000 mgr.smithi143.bissbc (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:16.945 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:16.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.798+0000 7f9ce359e700 1 -- 172.21.15.173:0/3079789572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ce4060180 msgr2=0x7f9ce40f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.798+0000 7f9ce359e700 1 --2- 172.21.15.173:0/3079789572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ce4060180 0x7f9ce40f8f80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f9ccc009fb0 tx=0x7f9ccc009b30).stop 2024-02-19T21:57:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.798+0000 7f9ce359e700 1 -- 172.21.15.173:0/3079789572 shutdown_connections 2024-02-19T21:57:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.798+0000 7f9ce359e700 1 --2- 172.21.15.173:0/3079789572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ce4060180 0x7f9ce40f8f80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.798+0000 7f9ce359e700 1 -- 172.21.15.173:0/3079789572 >> 172.21.15.173:0/3079789572 conn(0x7f9ce40f09a0 msgr2=0x7f9ce40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.799+0000 7f9ce359e700 1 -- 172.21.15.173:0/3079789572 shutdown_connections 2024-02-19T21:57:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.799+0000 7f9ce359e700 1 -- 172.21.15.173:0/3079789572 wait complete. 2024-02-19T21:57:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.799+0000 7f9ce359e700 1 Processor -- start 2024-02-19T21:57:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.799+0000 7f9ce359e700 1 -- start start 2024-02-19T21:57:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.799+0000 7f9ce359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ce4060180 0x7f9ce41065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.799+0000 7f9ce359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4106ad0 con 0x7f9ce4060180 2024-02-19T21:57:18.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.799+0000 7f9ce259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ce4060180 0x7f9ce41065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:18.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.799+0000 7f9ce259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ce4060180 0x7f9ce41065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:45316/0 (socket says 172.21.15.173:45316) 2024-02-19T21:57:18.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.799+0000 7f9ce259c700 1 -- 172.21.15.173:0/1972733687 learned_addr learned my addr 172.21.15.173:0/1972733687 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:18.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.800+0000 7f9ce259c700 1 -- 172.21.15.173:0/1972733687 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc005040 con 0x7f9ce4060180 2024-02-19T21:57:18.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.800+0000 7f9ce259c700 1 --2- 172.21.15.173:0/1972733687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ce4060180 0x7f9ce41065e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f9ccc005010 tx=0x7f9ccc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:18.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.800+0000 7f9cdaffd700 1 -- 172.21.15.173:0/1972733687 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ccc009320 con 0x7f9ce4060180 2024-02-19T21:57:18.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.800+0000 7f9ce359e700 1 -- 172.21.15.173:0/1972733687 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce4106cd0 con 0x7f9ce4060180 2024-02-19T21:57:18.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.800+0000 7f9cdaffd700 1 -- 172.21.15.173:0/1972733687 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ccc013380 con 0x7f9ce4060180 2024-02-19T21:57:18.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.800+0000 7f9cdaffd700 1 -- 172.21.15.173:0/1972733687 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ccc014e50 con 0x7f9ce4060180 2024-02-19T21:57:18.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.800+0000 7f9ce359e700 1 -- 172.21.15.173:0/1972733687 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce4107160 con 0x7f9ce4060180 2024-02-19T21:57:18.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.801+0000 7f9cdaffd700 1 -- 172.21.15.173:0/1972733687 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ccc014420 con 0x7f9ce4060180 2024-02-19T21:57:18.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.801+0000 7f9ce359e700 1 -- 172.21.15.173:0/1972733687 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce4045ad0 con 0x7f9ce4060180 2024-02-19T21:57:18.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.801+0000 7f9cdaffd700 1 --2- 172.21.15.173:0/1972733687 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9cd002ff10 0x7f9cd00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:18.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.801+0000 7f9cdaffd700 1 -- 172.21.15.173:0/1972733687 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ccc040e30 con 0x7f9ce4060180 2024-02-19T21:57:18.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.801+0000 7f9ce1d9b700 1 --2- 172.21.15.173:0/1972733687 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9cd002ff10 0x7f9cd00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:18.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.802+0000 7f9ce1d9b700 1 --2- 172.21.15.173:0/1972733687 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9cd002ff10 0x7f9cd00323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f9cd40096a0 tx=0x7f9cd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:18.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:18.806+0000 7f9cdaffd700 1 -- 172.21.15.173:0/1972733687 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ccc0095f0 con 0x7f9ce4060180 2024-02-19T21:57:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:16.465139+0000 mgr.smithi143.bissbc (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:19.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.016+0000 7f9ce359e700 1 -- 172.21.15.173:0/1972733687 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ce4108010 con 0x7f9ce4060180 2024-02-19T21:57:19.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.016+0000 7f9cdaffd700 1 -- 172.21.15.173:0/1972733687 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ccc01db70 con 0x7f9ce4060180 2024-02-19T21:57:19.016 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:19.016 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:19.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.018+0000 7f9ce359e700 1 -- 172.21.15.173:0/1972733687 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9cd002ff10 msgr2=0x7f9cd00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:19.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.018+0000 7f9ce359e700 1 --2- 172.21.15.173:0/1972733687 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9cd002ff10 0x7f9cd00323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f9cd40096a0 tx=0x7f9cd4006b40).stop 2024-02-19T21:57:19.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.018+0000 7f9ce359e700 1 -- 172.21.15.173:0/1972733687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ce4060180 msgr2=0x7f9ce41065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:19.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.018+0000 7f9ce359e700 1 --2- 172.21.15.173:0/1972733687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ce4060180 0x7f9ce41065e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f9ccc005010 tx=0x7f9ccc00b040).stop 2024-02-19T21:57:19.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.018+0000 7f9ce359e700 1 -- 172.21.15.173:0/1972733687 shutdown_connections 2024-02-19T21:57:19.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.018+0000 7f9ce359e700 1 --2- 172.21.15.173:0/1972733687 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9cd002ff10 0x7f9cd00323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:19.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.018+0000 7f9ce359e700 1 --2- 172.21.15.173:0/1972733687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ce4060180 0x7f9ce41065e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:19.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.018+0000 7f9ce359e700 1 -- 172.21.15.173:0/1972733687 >> 172.21.15.173:0/1972733687 conn(0x7f9ce40f09a0 msgr2=0x7f9ce40f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:19.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.018+0000 7f9ce359e700 1 -- 172.21.15.173:0/1972733687 shutdown_connections 2024-02-19T21:57:19.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:19.018+0000 7f9ce359e700 1 -- 172.21.15.173:0/1972733687 wait complete. 2024-02-19T21:57:19.019 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:19.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:19 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:19.017796+0000 mon.smithi143 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.173:0/1972733687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:20.758 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:20.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:18.465498+0000 mgr.smithi143.bissbc (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:20.562698+0000 mon.smithi143 (mon.0) 159 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:57:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:20.563094+0000 mon.smithi143 (mon.0) 160 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T21:57:22.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.627+0000 7f20a68ce700 1 -- 172.21.15.173:0/1637290406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20a00f6510 msgr2=0x7f20a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.627+0000 7f20a68ce700 1 --2- 172.21.15.173:0/1637290406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20a00f6510 0x7f20a00f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f2088009fb0 tx=0x7f2088009b30).stop 2024-02-19T21:57:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.627+0000 7f20a68ce700 1 -- 172.21.15.173:0/1637290406 shutdown_connections 2024-02-19T21:57:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.627+0000 7f20a68ce700 1 --2- 172.21.15.173:0/1637290406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20a00f6510 0x7f20a00f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.627+0000 7f20a68ce700 1 -- 172.21.15.173:0/1637290406 >> 172.21.15.173:0/1637290406 conn(0x7f20a00f1c70 msgr2=0x7f20a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.627+0000 7f20a68ce700 1 -- 172.21.15.173:0/1637290406 shutdown_connections 2024-02-19T21:57:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.627+0000 7f20a68ce700 1 -- 172.21.15.173:0/1637290406 wait complete. 2024-02-19T21:57:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.627+0000 7f20a68ce700 1 Processor -- start 2024-02-19T21:57:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.627+0000 7f20a68ce700 1 -- start start 2024-02-19T21:57:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.628+0000 7f20a68ce700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20a00f6510 0x7f20a006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.628+0000 7f20a68ce700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a0069e20 con 0x7f20a00f6510 2024-02-19T21:57:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.628+0000 7f209ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20a00f6510 0x7f20a006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.628+0000 7f209ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20a00f6510 0x7f20a006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:44306/0 (socket says 172.21.15.173:44306) 2024-02-19T21:57:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.628+0000 7f209ffff700 1 -- 172.21.15.173:0/518093580 learned_addr learned my addr 172.21.15.173:0/518093580 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.628+0000 7f209ffff700 1 -- 172.21.15.173:0/518093580 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2088005040 con 0x7f20a00f6510 2024-02-19T21:57:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.628+0000 7f209ffff700 1 --2- 172.21.15.173:0/518093580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20a00f6510 0x7f20a006b9a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2088005010 tx=0x7f208800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.628+0000 7f209cff9700 1 -- 172.21.15.173:0/518093580 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2088009320 con 0x7f20a00f6510 2024-02-19T21:57:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.628+0000 7f20a68ce700 1 -- 172.21.15.173:0/518093580 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a0069fc0 con 0x7f20a00f6510 2024-02-19T21:57:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.629+0000 7f209cff9700 1 -- 172.21.15.173:0/518093580 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2088013380 con 0x7f20a00f6510 2024-02-19T21:57:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.629+0000 7f20a68ce700 1 -- 172.21.15.173:0/518093580 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a006a450 con 0x7f20a00f6510 2024-02-19T21:57:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.629+0000 7f209cff9700 1 -- 172.21.15.173:0/518093580 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2088014e50 con 0x7f20a00f6510 2024-02-19T21:57:22.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.629+0000 7f209cff9700 1 -- 172.21.15.173:0/518093580 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20880263a0 con 0x7f20a00f6510 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.629+0000 7f209cff9700 1 --2- 172.21.15.173:0/518093580 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f208c02ff10 0x7f208c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.629+0000 7f209cff9700 1 -- 172.21.15.173:0/518093580 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20880095f0 con 0x7f20a00f6510 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.630+0000 7f20a68ce700 1 -- 172.21.15.173:0/518093580 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a0059100 con 0x7f20a00f6510 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.630+0000 7f209f7fe700 1 --2- 172.21.15.173:0/518093580 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f208c02ff10 0x7f208c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.630+0000 7f209f7fe700 1 --2- 172.21.15.173:0/518093580 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f208c02ff10 0x7f208c0323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f20900096a0 tx=0x7f2090006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:22.634 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.635+0000 7f209cff9700 1 -- 172.21.15.173:0/518093580 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2088012920 con 0x7f20a00f6510 2024-02-19T21:57:22.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.838+0000 7f20a68ce700 1 -- 172.21.15.173:0/518093580 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20a0045ad0 con 0x7f20a00f6510 2024-02-19T21:57:22.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.839+0000 7f209cff9700 1 -- 172.21.15.173:0/518093580 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f208802b030 con 0x7f20a00f6510 2024-02-19T21:57:22.839 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:22.839 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:22.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.840+0000 7f20a68ce700 1 -- 172.21.15.173:0/518093580 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f208c02ff10 msgr2=0x7f208c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:22.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.840+0000 7f20a68ce700 1 --2- 172.21.15.173:0/518093580 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f208c02ff10 0x7f208c0323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f20900096a0 tx=0x7f2090006b40).stop 2024-02-19T21:57:22.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.840+0000 7f20a68ce700 1 -- 172.21.15.173:0/518093580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20a00f6510 msgr2=0x7f20a006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:22.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.840+0000 7f20a68ce700 1 --2- 172.21.15.173:0/518093580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20a00f6510 0x7f20a006b9a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2088005010 tx=0x7f208800b040).stop 2024-02-19T21:57:22.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.840+0000 7f20a68ce700 1 -- 172.21.15.173:0/518093580 shutdown_connections 2024-02-19T21:57:22.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.840+0000 7f20a68ce700 1 --2- 172.21.15.173:0/518093580 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f208c02ff10 0x7f208c0323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:22.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.840+0000 7f20a68ce700 1 --2- 172.21.15.173:0/518093580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20a00f6510 0x7f20a006b9a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:22.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.840+0000 7f20a68ce700 1 -- 172.21.15.173:0/518093580 >> 172.21.15.173:0/518093580 conn(0x7f20a00f1c70 msgr2=0x7f20a00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:22.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.840+0000 7f20a68ce700 1 -- 172.21.15.173:0/518093580 shutdown_connections 2024-02-19T21:57:22.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:22.840+0000 7f20a68ce700 1 -- 172.21.15.173:0/518093580 wait complete. 2024-02-19T21:57:22.841 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:20.465871+0000 mgr.smithi143.bissbc (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:23.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:22.839796+0000 mon.smithi143 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.173:0/518093580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:24.529 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:24.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:24.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:22.466178+0000 mgr.smithi143.bissbc (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:26.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.507+0000 7f9da1c48700 1 -- 172.21.15.173:0/667600475 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d9c0f6510 msgr2=0x7f9d9c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:26.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.507+0000 7f9da1c48700 1 --2- 172.21.15.173:0/667600475 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d9c0f6510 0x7f9d9c0f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f9d8c009fb0 tx=0x7f9d8c009b30).stop 2024-02-19T21:57:26.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.507+0000 7f9da1c48700 1 -- 172.21.15.173:0/667600475 shutdown_connections 2024-02-19T21:57:26.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.507+0000 7f9da1c48700 1 --2- 172.21.15.173:0/667600475 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d9c0f6510 0x7f9d9c0f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:26.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.507+0000 7f9da1c48700 1 -- 172.21.15.173:0/667600475 >> 172.21.15.173:0/667600475 conn(0x7f9d9c0f1c70 msgr2=0x7f9d9c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:26.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.507+0000 7f9da1c48700 1 -- 172.21.15.173:0/667600475 shutdown_connections 2024-02-19T21:57:26.509 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.507+0000 7f9da1c48700 1 -- 172.21.15.173:0/667600475 wait complete. 2024-02-19T21:57:26.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.508+0000 7f9da1c48700 1 Processor -- start 2024-02-19T21:57:26.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.508+0000 7f9da1c48700 1 -- start start 2024-02-19T21:57:26.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.508+0000 7f9da1c48700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d9c0f6510 0x7f9d9c1089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:26.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.508+0000 7f9da1c48700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c108eb0 con 0x7f9d9c0f6510 2024-02-19T21:57:26.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.508+0000 7f9d9b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d9c0f6510 0x7f9d9c1089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:26.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.508+0000 7f9d9b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d9c0f6510 0x7f9d9c1089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:44322/0 (socket says 172.21.15.173:44322) 2024-02-19T21:57:26.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.508+0000 7f9d9b7fe700 1 -- 172.21.15.173:0/1499531313 learned_addr learned my addr 172.21.15.173:0/1499531313 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:26.511 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.509+0000 7f9d9b7fe700 1 -- 172.21.15.173:0/1499531313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c005040 con 0x7f9d9c0f6510 2024-02-19T21:57:26.511 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.509+0000 7f9d9b7fe700 1 --2- 172.21.15.173:0/1499531313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d9c0f6510 0x7f9d9c1089c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f9d8c005010 tx=0x7f9d8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:26.511 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.509+0000 7f9d83fff700 1 -- 172.21.15.173:0/1499531313 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d8c009320 con 0x7f9d9c0f6510 2024-02-19T21:57:26.511 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.509+0000 7f9da1c48700 1 -- 172.21.15.173:0/1499531313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c1090b0 con 0x7f9d9c0f6510 2024-02-19T21:57:26.511 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.509+0000 7f9d83fff700 1 -- 172.21.15.173:0/1499531313 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d8c013380 con 0x7f9d9c0f6510 2024-02-19T21:57:26.511 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.509+0000 7f9d83fff700 1 -- 172.21.15.173:0/1499531313 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d8c014e50 con 0x7f9d9c0f6510 2024-02-19T21:57:26.511 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.509+0000 7f9da1c48700 1 -- 172.21.15.173:0/1499531313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c109540 con 0x7f9d9c0f6510 2024-02-19T21:57:26.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.510+0000 7f9d83fff700 1 -- 172.21.15.173:0/1499531313 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d8c0263a0 con 0x7f9d9c0f6510 2024-02-19T21:57:26.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.510+0000 7f9d83fff700 1 --2- 172.21.15.173:0/1499531313 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d8402ff10 0x7f9d840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:26.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.510+0000 7f9d83fff700 1 -- 172.21.15.173:0/1499531313 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d8c040f50 con 0x7f9d9c0f6510 2024-02-19T21:57:26.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.510+0000 7f9d9affd700 1 --2- 172.21.15.173:0/1499531313 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d8402ff10 0x7f9d840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:26.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.510+0000 7f9da1c48700 1 -- 172.21.15.173:0/1499531313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c059100 con 0x7f9d9c0f6510 2024-02-19T21:57:26.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.511+0000 7f9d9affd700 1 --2- 172.21.15.173:0/1499531313 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d8402ff10 0x7f9d840323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f9d900096a0 tx=0x7f9d90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:26.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.515+0000 7f9d83fff700 1 -- 172.21.15.173:0/1499531313 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d8c012920 con 0x7f9d9c0f6510 2024-02-19T21:57:26.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.719+0000 7f9da1c48700 1 -- 172.21.15.173:0/1499531313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d9c06b3c0 con 0x7f9d9c0f6510 2024-02-19T21:57:26.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.720+0000 7f9d83fff700 1 -- 172.21.15.173:0/1499531313 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d8c02b020 con 0x7f9d9c0f6510 2024-02-19T21:57:26.720 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:26.720 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:26.721 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.722+0000 7f9da1c48700 1 -- 172.21.15.173:0/1499531313 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d8402ff10 msgr2=0x7f9d840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:26.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.722+0000 7f9da1c48700 1 --2- 172.21.15.173:0/1499531313 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d8402ff10 0x7f9d840323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f9d900096a0 tx=0x7f9d90006b40).stop 2024-02-19T21:57:26.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.722+0000 7f9da1c48700 1 -- 172.21.15.173:0/1499531313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d9c0f6510 msgr2=0x7f9d9c1089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:26.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.722+0000 7f9da1c48700 1 --2- 172.21.15.173:0/1499531313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d9c0f6510 0x7f9d9c1089c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f9d8c005010 tx=0x7f9d8c00b040).stop 2024-02-19T21:57:26.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.722+0000 7f9da1c48700 1 -- 172.21.15.173:0/1499531313 shutdown_connections 2024-02-19T21:57:26.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.722+0000 7f9da1c48700 1 --2- 172.21.15.173:0/1499531313 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d8402ff10 0x7f9d840323d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:26.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.722+0000 7f9da1c48700 1 --2- 172.21.15.173:0/1499531313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d9c0f6510 0x7f9d9c1089c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:26.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.722+0000 7f9da1c48700 1 -- 172.21.15.173:0/1499531313 >> 172.21.15.173:0/1499531313 conn(0x7f9d9c0f1c70 msgr2=0x7f9d9c0f2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:26.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.722+0000 7f9da1c48700 1 -- 172.21.15.173:0/1499531313 shutdown_connections 2024-02-19T21:57:26.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:26.722+0000 7f9da1c48700 1 -- 172.21.15.173:0/1499531313 wait complete. 2024-02-19T21:57:26.723 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:24.466454+0000 mgr.smithi143.bissbc (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:27.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:27 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:26.721236+0000 mon.smithi143 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.173:0/1499531313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:28.453 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:28.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:26.466693+0000 mgr.smithi143.bissbc (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:30.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.431+0000 7fdf1d53d700 1 -- 172.21.15.173:0/2307996785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf180601e0 msgr2=0x7fdf180f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:30.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.431+0000 7fdf1d53d700 1 --2- 172.21.15.173:0/2307996785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf180601e0 0x7fdf180f8f20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fdf08009fb0 tx=0x7fdf08009b30).stop 2024-02-19T21:57:30.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.432+0000 7fdf1d53d700 1 -- 172.21.15.173:0/2307996785 shutdown_connections 2024-02-19T21:57:30.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.432+0000 7fdf1d53d700 1 --2- 172.21.15.173:0/2307996785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf180601e0 0x7fdf180f8f20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:30.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.432+0000 7fdf1d53d700 1 -- 172.21.15.173:0/2307996785 >> 172.21.15.173:0/2307996785 conn(0x7fdf180f09a0 msgr2=0x7fdf180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:30.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.432+0000 7fdf1d53d700 1 -- 172.21.15.173:0/2307996785 shutdown_connections 2024-02-19T21:57:30.432 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.432+0000 7fdf1d53d700 1 -- 172.21.15.173:0/2307996785 wait complete. 2024-02-19T21:57:30.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.432+0000 7fdf1d53d700 1 Processor -- start 2024-02-19T21:57:30.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.432+0000 7fdf1d53d700 1 -- start start 2024-02-19T21:57:30.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.432+0000 7fdf1d53d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf180601e0 0x7fdf18106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:30.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.432+0000 7fdf1d53d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf18106af0 con 0x7fdf180601e0 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.433+0000 7fdf17fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf180601e0 0x7fdf18106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.433+0000 7fdf17fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf180601e0 0x7fdf18106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:44330/0 (socket says 172.21.15.173:44330) 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.433+0000 7fdf17fff700 1 -- 172.21.15.173:0/1659888702 learned_addr learned my addr 172.21.15.173:0/1659888702 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.433+0000 7fdf17fff700 1 -- 172.21.15.173:0/1659888702 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf08005040 con 0x7fdf180601e0 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.433+0000 7fdf17fff700 1 --2- 172.21.15.173:0/1659888702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf180601e0 0x7fdf18106600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdf08005010 tx=0x7fdf0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.433+0000 7fdf14ff9700 1 -- 172.21.15.173:0/1659888702 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf08009320 con 0x7fdf180601e0 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.433+0000 7fdf1d53d700 1 -- 172.21.15.173:0/1659888702 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf18106cf0 con 0x7fdf180601e0 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.433+0000 7fdf14ff9700 1 -- 172.21.15.173:0/1659888702 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf08013380 con 0x7fdf180601e0 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.433+0000 7fdf14ff9700 1 -- 172.21.15.173:0/1659888702 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf08014e50 con 0x7fdf180601e0 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.433+0000 7fdf1d53d700 1 -- 172.21.15.173:0/1659888702 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf18107180 con 0x7fdf180601e0 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.434+0000 7fdf14ff9700 1 -- 172.21.15.173:0/1659888702 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf08014420 con 0x7fdf180601e0 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.434+0000 7fdf1d53d700 1 -- 172.21.15.173:0/1659888702 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf18045ad0 con 0x7fdf180601e0 2024-02-19T21:57:30.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.434+0000 7fdf14ff9700 1 --2- 172.21.15.173:0/1659888702 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdf0002ff10 0x7fdf000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:30.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.434+0000 7fdf14ff9700 1 -- 172.21.15.173:0/1659888702 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf08040f40 con 0x7fdf180601e0 2024-02-19T21:57:30.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.434+0000 7fdf177fe700 1 --2- 172.21.15.173:0/1659888702 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdf0002ff10 0x7fdf000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:30.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.435+0000 7fdf177fe700 1 --2- 172.21.15.173:0/1659888702 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdf0002ff10 0x7fdf000323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fdf0c0096a0 tx=0x7fdf0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:30.439 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.439+0000 7fdf14ff9700 1 -- 172.21.15.173:0/1659888702 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf080095f0 con 0x7fdf180601e0 2024-02-19T21:57:30.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.643+0000 7fdf1d53d700 1 -- 172.21.15.173:0/1659888702 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf18108030 con 0x7fdf180601e0 2024-02-19T21:57:30.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.644+0000 7fdf14ff9700 1 -- 172.21.15.173:0/1659888702 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdf0801db70 con 0x7fdf180601e0 2024-02-19T21:57:30.643 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:30.644 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:30.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.645+0000 7fdf1d53d700 1 -- 172.21.15.173:0/1659888702 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdf0002ff10 msgr2=0x7fdf000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:30.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.645+0000 7fdf1d53d700 1 --2- 172.21.15.173:0/1659888702 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdf0002ff10 0x7fdf000323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fdf0c0096a0 tx=0x7fdf0c006b40).stop 2024-02-19T21:57:30.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.645+0000 7fdf1d53d700 1 -- 172.21.15.173:0/1659888702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf180601e0 msgr2=0x7fdf18106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:30.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.645+0000 7fdf1d53d700 1 --2- 172.21.15.173:0/1659888702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf180601e0 0x7fdf18106600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdf08005010 tx=0x7fdf0800b040).stop 2024-02-19T21:57:30.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.645+0000 7fdf1d53d700 1 -- 172.21.15.173:0/1659888702 shutdown_connections 2024-02-19T21:57:30.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.645+0000 7fdf1d53d700 1 --2- 172.21.15.173:0/1659888702 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdf0002ff10 0x7fdf000323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:30.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.645+0000 7fdf1d53d700 1 --2- 172.21.15.173:0/1659888702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf180601e0 0x7fdf18106600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:30.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.645+0000 7fdf1d53d700 1 -- 172.21.15.173:0/1659888702 >> 172.21.15.173:0/1659888702 conn(0x7fdf180f09a0 msgr2=0x7fdf180f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:30.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.645+0000 7fdf1d53d700 1 -- 172.21.15.173:0/1659888702 shutdown_connections 2024-02-19T21:57:30.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:30.645+0000 7fdf1d53d700 1 -- 172.21.15.173:0/1659888702 wait complete. 2024-02-19T21:57:30.646 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:28.466960+0000 mgr.smithi143.bissbc (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:31.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:31 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:30.467335+0000 mgr.smithi143.bissbc (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:31.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:31 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:30.644702+0000 mon.smithi143 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.173:0/1659888702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:32.236 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:32.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:34.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.150+0000 7fb9704bb700 1 -- 172.21.15.173:0/465542809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9680f5410 msgr2=0x7fb9680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.150+0000 7fb9704bb700 1 --2- 172.21.15.173:0/465542809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9680f5410 0x7fb9680f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fb964009fb0 tx=0x7fb964009b30).stop 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.150+0000 7fb9704bb700 1 -- 172.21.15.173:0/465542809 shutdown_connections 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.150+0000 7fb9704bb700 1 --2- 172.21.15.173:0/465542809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9680f5410 0x7fb9680f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.150+0000 7fb9704bb700 1 -- 172.21.15.173:0/465542809 >> 172.21.15.173:0/465542809 conn(0x7fb9680f0b70 msgr2=0x7fb9680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.150+0000 7fb9704bb700 1 -- 172.21.15.173:0/465542809 shutdown_connections 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.150+0000 7fb9704bb700 1 -- 172.21.15.173:0/465542809 wait complete. 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.151+0000 7fb9704bb700 1 Processor -- start 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.151+0000 7fb9704bb700 1 -- start start 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.151+0000 7fb9704bb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9680f5410 0x7fb9681094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.151+0000 7fb9704bb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb968060240 con 0x7fb9680f5410 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.151+0000 7fb96e257700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9680f5410 0x7fb9681094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.151+0000 7fb96e257700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9680f5410 0x7fb9681094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:45098/0 (socket says 172.21.15.173:45098) 2024-02-19T21:57:34.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.151+0000 7fb96e257700 1 -- 172.21.15.173:0/932910917 learned_addr learned my addr 172.21.15.173:0/932910917 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:34.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.151+0000 7fb96e257700 1 -- 172.21.15.173:0/932910917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb964005040 con 0x7fb9680f5410 2024-02-19T21:57:34.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.151+0000 7fb96e257700 1 --2- 172.21.15.173:0/932910917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9680f5410 0x7fb9681094e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb964005010 tx=0x7fb96400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:34.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.152+0000 7fb95effd700 1 -- 172.21.15.173:0/932910917 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb964009320 con 0x7fb9680f5410 2024-02-19T21:57:34.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.152+0000 7fb9704bb700 1 -- 172.21.15.173:0/932910917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb968060440 con 0x7fb9680f5410 2024-02-19T21:57:34.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.152+0000 7fb95effd700 1 -- 172.21.15.173:0/932910917 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb964013380 con 0x7fb9680f5410 2024-02-19T21:57:34.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.152+0000 7fb95effd700 1 -- 172.21.15.173:0/932910917 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb964014e50 con 0x7fb9680f5410 2024-02-19T21:57:34.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.152+0000 7fb9704bb700 1 -- 172.21.15.173:0/932910917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb968109b90 con 0x7fb9680f5410 2024-02-19T21:57:34.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.152+0000 7fb95effd700 1 -- 172.21.15.173:0/932910917 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb964014420 con 0x7fb9680f5410 2024-02-19T21:57:34.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.152+0000 7fb9704bb700 1 -- 172.21.15.173:0/932910917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb968045ad0 con 0x7fb9680f5410 2024-02-19T21:57:34.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.152+0000 7fb95effd700 1 --2- 172.21.15.173:0/932910917 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb95402ff10 0x7fb9540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:34.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.153+0000 7fb95effd700 1 -- 172.21.15.173:0/932910917 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb964040f70 con 0x7fb9680f5410 2024-02-19T21:57:34.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.153+0000 7fb96da56700 1 --2- 172.21.15.173:0/932910917 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb95402ff10 0x7fb9540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:34.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.153+0000 7fb96da56700 1 --2- 172.21.15.173:0/932910917 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb95402ff10 0x7fb9540323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb9580096a0 tx=0x7fb958006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:34.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.158+0000 7fb95effd700 1 -- 172.21.15.173:0/932910917 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9640095f0 con 0x7fb9680f5410 2024-02-19T21:57:34.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.362+0000 7fb9704bb700 1 -- 172.21.15.173:0/932910917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb96810a410 con 0x7fb9680f5410 2024-02-19T21:57:34.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.362+0000 7fb95effd700 1 -- 172.21.15.173:0/932910917 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb96401db70 con 0x7fb9680f5410 2024-02-19T21:57:34.362 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:34.362 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:34.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.364+0000 7fb9704bb700 1 -- 172.21.15.173:0/932910917 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb95402ff10 msgr2=0x7fb9540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:34.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.364+0000 7fb9704bb700 1 --2- 172.21.15.173:0/932910917 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb95402ff10 0x7fb9540323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb9580096a0 tx=0x7fb958006b40).stop 2024-02-19T21:57:34.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.364+0000 7fb9704bb700 1 -- 172.21.15.173:0/932910917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9680f5410 msgr2=0x7fb9681094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:34.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.364+0000 7fb9704bb700 1 --2- 172.21.15.173:0/932910917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9680f5410 0x7fb9681094e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb964005010 tx=0x7fb96400b040).stop 2024-02-19T21:57:34.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.364+0000 7fb9704bb700 1 -- 172.21.15.173:0/932910917 shutdown_connections 2024-02-19T21:57:34.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.364+0000 7fb9704bb700 1 --2- 172.21.15.173:0/932910917 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb95402ff10 0x7fb9540323d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:34.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.364+0000 7fb9704bb700 1 --2- 172.21.15.173:0/932910917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9680f5410 0x7fb9681094e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:34.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.364+0000 7fb9704bb700 1 -- 172.21.15.173:0/932910917 >> 172.21.15.173:0/932910917 conn(0x7fb9680f0b70 msgr2=0x7fb9680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:34.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.364+0000 7fb9704bb700 1 -- 172.21.15.173:0/932910917 shutdown_connections 2024-02-19T21:57:34.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:34.364+0000 7fb9704bb700 1 -- 172.21.15.173:0/932910917 wait complete. 2024-02-19T21:57:34.365 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:32.467715+0000 mgr.smithi143.bissbc (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:34.363567+0000 mon.smithi143 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.173:0/932910917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:36.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:36.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:34.468043+0000 mgr.smithi143.bissbc (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:37.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.962+0000 7fcd25872700 1 -- 172.21.15.173:0/230874945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd200601e0 msgr2=0x7fcd200f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:37.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.962+0000 7fcd25872700 1 --2- 172.21.15.173:0/230874945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd200601e0 0x7fcd200f9190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fcd10009fb0 tx=0x7fcd10009b30).stop 2024-02-19T21:57:37.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.962+0000 7fcd25872700 1 -- 172.21.15.173:0/230874945 shutdown_connections 2024-02-19T21:57:37.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.962+0000 7fcd25872700 1 --2- 172.21.15.173:0/230874945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd200601e0 0x7fcd200f9190 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:37.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.962+0000 7fcd25872700 1 -- 172.21.15.173:0/230874945 >> 172.21.15.173:0/230874945 conn(0x7fcd200f0b70 msgr2=0x7fcd200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:37.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.962+0000 7fcd25872700 1 -- 172.21.15.173:0/230874945 shutdown_connections 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.962+0000 7fcd25872700 1 -- 172.21.15.173:0/230874945 wait complete. 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.963+0000 7fcd25872700 1 Processor -- start 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.963+0000 7fcd25872700 1 -- start start 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.963+0000 7fcd25872700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd200601e0 0x7fcd201067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.963+0000 7fcd25872700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd20106cc0 con 0x7fcd200601e0 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.963+0000 7fcd1effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd200601e0 0x7fcd201067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.963+0000 7fcd1effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd200601e0 0x7fcd201067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:45112/0 (socket says 172.21.15.173:45112) 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.963+0000 7fcd1effd700 1 -- 172.21.15.173:0/865516827 learned_addr learned my addr 172.21.15.173:0/865516827 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.964+0000 7fcd1effd700 1 -- 172.21.15.173:0/865516827 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd10005040 con 0x7fcd200601e0 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.964+0000 7fcd1effd700 1 --2- 172.21.15.173:0/865516827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd200601e0 0x7fcd201067d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fcd10007a10 tx=0x7fcd10009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.964+0000 7fcd077fe700 1 -- 172.21.15.173:0/865516827 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd10018070 con 0x7fcd200601e0 2024-02-19T21:57:37.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.964+0000 7fcd25872700 1 -- 172.21.15.173:0/865516827 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd20106ec0 con 0x7fcd200601e0 2024-02-19T21:57:37.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.964+0000 7fcd077fe700 1 -- 172.21.15.173:0/865516827 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd10013410 con 0x7fcd200601e0 2024-02-19T21:57:37.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.964+0000 7fcd077fe700 1 -- 172.21.15.173:0/865516827 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd1001c3c0 con 0x7fcd200601e0 2024-02-19T21:57:37.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.964+0000 7fcd25872700 1 -- 172.21.15.173:0/865516827 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd20107350 con 0x7fcd200601e0 2024-02-19T21:57:37.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.965+0000 7fcd077fe700 1 -- 172.21.15.173:0/865516827 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd10013ca0 con 0x7fcd200601e0 2024-02-19T21:57:37.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.965+0000 7fcd25872700 1 -- 172.21.15.173:0/865516827 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd20045ad0 con 0x7fcd200601e0 2024-02-19T21:57:37.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.965+0000 7fcd077fe700 1 --2- 172.21.15.173:0/865516827 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0802ff60 0x7fcd08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:37.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.965+0000 7fcd077fe700 1 -- 172.21.15.173:0/865516827 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd10041510 con 0x7fcd200601e0 2024-02-19T21:57:37.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.965+0000 7fcd1e7fc700 1 --2- 172.21.15.173:0/865516827 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0802ff60 0x7fcd08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:37.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.965+0000 7fcd1e7fc700 1 --2- 172.21.15.173:0/865516827 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0802ff60 0x7fcd08032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fcd140096a0 tx=0x7fcd14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:37.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:37.970+0000 7fcd077fe700 1 -- 172.21.15.173:0/865516827 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd1001d420 con 0x7fcd200601e0 2024-02-19T21:57:38.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.176+0000 7fcd25872700 1 -- 172.21.15.173:0/865516827 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd201081a0 con 0x7fcd200601e0 2024-02-19T21:57:38.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.177+0000 7fcd077fe700 1 -- 172.21.15.173:0/865516827 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd10025ce0 con 0x7fcd200601e0 2024-02-19T21:57:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:38.177 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:38.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.178+0000 7fcd25872700 1 -- 172.21.15.173:0/865516827 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0802ff60 msgr2=0x7fcd08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:38.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.178+0000 7fcd25872700 1 --2- 172.21.15.173:0/865516827 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0802ff60 0x7fcd08032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fcd140096a0 tx=0x7fcd14006b40).stop 2024-02-19T21:57:38.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.178+0000 7fcd25872700 1 -- 172.21.15.173:0/865516827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd200601e0 msgr2=0x7fcd201067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:38.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.178+0000 7fcd25872700 1 --2- 172.21.15.173:0/865516827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd200601e0 0x7fcd201067d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fcd10007a10 tx=0x7fcd10009b30).stop 2024-02-19T21:57:38.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.178+0000 7fcd25872700 1 -- 172.21.15.173:0/865516827 shutdown_connections 2024-02-19T21:57:38.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.178+0000 7fcd25872700 1 --2- 172.21.15.173:0/865516827 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0802ff60 0x7fcd08032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:38.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.178+0000 7fcd25872700 1 --2- 172.21.15.173:0/865516827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd200601e0 0x7fcd201067d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:38.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.178+0000 7fcd25872700 1 -- 172.21.15.173:0/865516827 >> 172.21.15.173:0/865516827 conn(0x7fcd200f0b70 msgr2=0x7fcd200f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:38.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.178+0000 7fcd25872700 1 -- 172.21.15.173:0/865516827 shutdown_connections 2024-02-19T21:57:38.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:38.178+0000 7fcd25872700 1 -- 172.21.15.173:0/865516827 wait complete. 2024-02-19T21:57:38.179 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:36.468387+0000 mgr.smithi143.bissbc (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:38.177837+0000 mon.smithi143 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.173:0/865516827' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:40.040 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:40.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:38.468672+0000 mgr.smithi143.bissbc (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.981+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/924462919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0f5410 msgr2=0x7f7ecc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.981+0000 7f7ed0f1c700 1 --2- 172.21.15.173:0/924462919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0f5410 0x7f7ecc0f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7eb4009fb0 tx=0x7f7eb4009b30).stop 2024-02-19T21:57:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.981+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/924462919 shutdown_connections 2024-02-19T21:57:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.981+0000 7f7ed0f1c700 1 --2- 172.21.15.173:0/924462919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0f5410 0x7f7ecc0f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.981+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/924462919 >> 172.21.15.173:0/924462919 conn(0x7f7ecc0f0b70 msgr2=0x7f7ecc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:41.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.981+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/924462919 shutdown_connections 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.982+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/924462919 wait complete. 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.982+0000 7f7ed0f1c700 1 Processor -- start 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.982+0000 7f7ed0f1c700 1 -- start start 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.982+0000 7f7ed0f1c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0f5410 0x7f7ecc1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.982+0000 7f7ed0f1c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc060240 con 0x7f7ecc0f5410 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.982+0000 7f7eca59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0f5410 0x7f7ecc1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.982+0000 7f7eca59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0f5410 0x7f7ecc1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:37550/0 (socket says 172.21.15.173:37550) 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.982+0000 7f7eca59c700 1 -- 172.21.15.173:0/3176383122 learned_addr learned my addr 172.21.15.173:0/3176383122 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.983+0000 7f7eca59c700 1 -- 172.21.15.173:0/3176383122 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4005040 con 0x7f7ecc0f5410 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.983+0000 7f7eca59c700 1 --2- 172.21.15.173:0/3176383122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0f5410 0x7f7ecc1094e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7eb4005010 tx=0x7f7eb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.983+0000 7f7ec2ffd700 1 -- 172.21.15.173:0/3176383122 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7eb4009320 con 0x7f7ecc0f5410 2024-02-19T21:57:41.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.983+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/3176383122 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ecc060440 con 0x7f7ecc0f5410 2024-02-19T21:57:41.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.983+0000 7f7ec2ffd700 1 -- 172.21.15.173:0/3176383122 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7eb4013380 con 0x7f7ecc0f5410 2024-02-19T21:57:41.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.983+0000 7f7ec2ffd700 1 -- 172.21.15.173:0/3176383122 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7eb4014e50 con 0x7f7ecc0f5410 2024-02-19T21:57:41.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.983+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/3176383122 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ecc109b90 con 0x7f7ecc0f5410 2024-02-19T21:57:41.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.984+0000 7f7ec2ffd700 1 -- 172.21.15.173:0/3176383122 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7eb40263a0 con 0x7f7ecc0f5410 2024-02-19T21:57:41.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.984+0000 7f7ec2ffd700 1 --2- 172.21.15.173:0/3176383122 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7eb802ff10 0x7f7eb80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:41.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.984+0000 7f7ec2ffd700 1 -- 172.21.15.173:0/3176383122 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7eb4040fa0 con 0x7f7ecc0f5410 2024-02-19T21:57:41.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.984+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/3176383122 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ecc045ad0 con 0x7f7ecc0f5410 2024-02-19T21:57:41.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.984+0000 7f7ec9d9b700 1 --2- 172.21.15.173:0/3176383122 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7eb802ff10 0x7f7eb80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:41.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.984+0000 7f7ec9d9b700 1 --2- 172.21.15.173:0/3176383122 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7eb802ff10 0x7f7eb80323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f7ebc0096a0 tx=0x7f7ebc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:41.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:41.989+0000 7f7ec2ffd700 1 -- 172.21.15.173:0/3176383122 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7eb4012920 con 0x7f7ecc0f5410 2024-02-19T21:57:42.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.193+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/3176383122 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ecc10a240 con 0x7f7ecc0f5410 2024-02-19T21:57:42.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.195+0000 7f7ec2ffd700 1 -- 172.21.15.173:0/3176383122 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7eb402b020 con 0x7f7ecc0f5410 2024-02-19T21:57:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:42.194 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:42.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.195+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/3176383122 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7eb802ff10 msgr2=0x7f7eb80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:42.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.195+0000 7f7ed0f1c700 1 --2- 172.21.15.173:0/3176383122 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7eb802ff10 0x7f7eb80323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f7ebc0096a0 tx=0x7f7ebc006b40).stop 2024-02-19T21:57:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.195+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/3176383122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0f5410 msgr2=0x7f7ecc1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.195+0000 7f7ed0f1c700 1 --2- 172.21.15.173:0/3176383122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0f5410 0x7f7ecc1094e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7eb4005010 tx=0x7f7eb400b040).stop 2024-02-19T21:57:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.195+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/3176383122 shutdown_connections 2024-02-19T21:57:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.195+0000 7f7ed0f1c700 1 --2- 172.21.15.173:0/3176383122 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7eb802ff10 0x7f7eb80323d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.195+0000 7f7ed0f1c700 1 --2- 172.21.15.173:0/3176383122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0f5410 0x7f7ecc1094e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.195+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/3176383122 >> 172.21.15.173:0/3176383122 conn(0x7f7ecc0f0b70 msgr2=0x7f7ecc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.195+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/3176383122 shutdown_connections 2024-02-19T21:57:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:42.195+0000 7f7ed0f1c700 1 -- 172.21.15.173:0/3176383122 wait complete. 2024-02-19T21:57:42.196 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:40.469003+0000 mgr.smithi143.bissbc (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:42.195022+0000 mon.smithi143 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.173:0/3176383122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:43.809 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:43.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:42.469412+0000 mgr.smithi143.bissbc (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:45.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.834+0000 7f5ea0233700 1 -- 172.21.15.173:0/486359953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e980f6510 msgr2=0x7f5e980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:45.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.834+0000 7f5ea0233700 1 --2- 172.21.15.173:0/486359953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e980f6510 0x7f5e980f6930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f5e88009fb0 tx=0x7f5e88009b30).stop 2024-02-19T21:57:45.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.834+0000 7f5ea0233700 1 -- 172.21.15.173:0/486359953 shutdown_connections 2024-02-19T21:57:45.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.834+0000 7f5ea0233700 1 --2- 172.21.15.173:0/486359953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e980f6510 0x7f5e980f6930 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:45.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.834+0000 7f5ea0233700 1 -- 172.21.15.173:0/486359953 >> 172.21.15.173:0/486359953 conn(0x7f5e980f1c70 msgr2=0x7f5e980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:45.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.834+0000 7f5ea0233700 1 -- 172.21.15.173:0/486359953 shutdown_connections 2024-02-19T21:57:45.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.834+0000 7f5ea0233700 1 -- 172.21.15.173:0/486359953 wait complete. 2024-02-19T21:57:45.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.835+0000 7f5ea0233700 1 Processor -- start 2024-02-19T21:57:45.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.835+0000 7f5ea0233700 1 -- start start 2024-02-19T21:57:45.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.835+0000 7f5ea0233700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e980f6510 0x7f5e9806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:45.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.835+0000 7f5ea0233700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e98069e20 con 0x7f5e980f6510 2024-02-19T21:57:45.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.835+0000 7f5e9dfcf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e980f6510 0x7f5e9806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:45.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.835+0000 7f5e9dfcf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e980f6510 0x7f5e9806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:37562/0 (socket says 172.21.15.173:37562) 2024-02-19T21:57:45.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.835+0000 7f5e9dfcf700 1 -- 172.21.15.173:0/453762452 learned_addr learned my addr 172.21.15.173:0/453762452 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:45.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.836+0000 7f5e9dfcf700 1 -- 172.21.15.173:0/453762452 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88005040 con 0x7f5e980f6510 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.836+0000 7f5e9dfcf700 1 --2- 172.21.15.173:0/453762452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e980f6510 0x7f5e9806b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f5e88005010 tx=0x7f5e8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.836+0000 7f5e8e7fc700 1 -- 172.21.15.173:0/453762452 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e88009320 con 0x7f5e980f6510 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.836+0000 7f5e8e7fc700 1 -- 172.21.15.173:0/453762452 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e88013380 con 0x7f5e980f6510 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.836+0000 7f5ea0233700 1 -- 172.21.15.173:0/453762452 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e98069fc0 con 0x7f5e980f6510 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.836+0000 7f5e8e7fc700 1 -- 172.21.15.173:0/453762452 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e88014e50 con 0x7f5e980f6510 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.836+0000 7f5ea0233700 1 -- 172.21.15.173:0/453762452 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9806a450 con 0x7f5e980f6510 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.837+0000 7f5e8e7fc700 1 -- 172.21.15.173:0/453762452 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e880263a0 con 0x7f5e980f6510 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.837+0000 7f5e8e7fc700 1 --2- 172.21.15.173:0/453762452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5e8402ff10 0x7f5e840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.837+0000 7f5ea0233700 1 -- 172.21.15.173:0/453762452 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e98059100 con 0x7f5e980f6510 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.837+0000 7f5e8e7fc700 1 -- 172.21.15.173:0/453762452 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e88040f00 con 0x7f5e980f6510 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.837+0000 7f5e9d7ce700 1 --2- 172.21.15.173:0/453762452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5e8402ff10 0x7f5e840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:45.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.838+0000 7f5e9d7ce700 1 --2- 172.21.15.173:0/453762452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5e8402ff10 0x7f5e840323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f5e940096a0 tx=0x7f5e94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:45.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:45.843+0000 7f5e8e7fc700 1 -- 172.21.15.173:0/453762452 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e88026630 con 0x7f5e980f6510 2024-02-19T21:57:46.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.048+0000 7f5ea0233700 1 -- 172.21.15.173:0/453762452 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e980fc0a0 con 0x7f5e980f6510 2024-02-19T21:57:46.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.050+0000 7f5e8e7fc700 1 -- 172.21.15.173:0/453762452 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5e8802b020 con 0x7f5e980f6510 2024-02-19T21:57:46.050 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:46.050 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:46.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.051+0000 7f5ea0233700 1 -- 172.21.15.173:0/453762452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5e8402ff10 msgr2=0x7f5e840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:46.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.051+0000 7f5ea0233700 1 --2- 172.21.15.173:0/453762452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5e8402ff10 0x7f5e840323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f5e940096a0 tx=0x7f5e94006b40).stop 2024-02-19T21:57:46.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.051+0000 7f5ea0233700 1 -- 172.21.15.173:0/453762452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e980f6510 msgr2=0x7f5e9806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:46.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.051+0000 7f5ea0233700 1 --2- 172.21.15.173:0/453762452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e980f6510 0x7f5e9806b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f5e88005010 tx=0x7f5e8800b040).stop 2024-02-19T21:57:46.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.051+0000 7f5ea0233700 1 -- 172.21.15.173:0/453762452 shutdown_connections 2024-02-19T21:57:46.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.051+0000 7f5ea0233700 1 --2- 172.21.15.173:0/453762452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f5e8402ff10 0x7f5e840323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:46.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.051+0000 7f5ea0233700 1 --2- 172.21.15.173:0/453762452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e980f6510 0x7f5e9806b9a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:46.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.051+0000 7f5ea0233700 1 -- 172.21.15.173:0/453762452 >> 172.21.15.173:0/453762452 conn(0x7f5e980f1c70 msgr2=0x7f5e980f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:46.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.051+0000 7f5ea0233700 1 -- 172.21.15.173:0/453762452 shutdown_connections 2024-02-19T21:57:46.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:46.051+0000 7f5ea0233700 1 -- 172.21.15.173:0/453762452 wait complete. 2024-02-19T21:57:46.053 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:44.469816+0000 mgr.smithi143.bissbc (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:46.051225+0000 mon.smithi143 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.173:0/453762452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:47.783 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:47.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:46.470020+0000 mgr.smithi143.bissbc (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:49.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.718+0000 7f07fec3d700 1 -- 172.21.15.173:0/235002910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07f80f5410 msgr2=0x7f07f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:49.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.718+0000 7f07fec3d700 1 --2- 172.21.15.173:0/235002910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07f80f5410 0x7f07f80f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f07ec009fb0 tx=0x7f07ec009b30).stop 2024-02-19T21:57:49.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.718+0000 7f07fec3d700 1 -- 172.21.15.173:0/235002910 shutdown_connections 2024-02-19T21:57:49.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.718+0000 7f07fec3d700 1 --2- 172.21.15.173:0/235002910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07f80f5410 0x7f07f80f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:49.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.718+0000 7f07fec3d700 1 -- 172.21.15.173:0/235002910 >> 172.21.15.173:0/235002910 conn(0x7f07f80f0b70 msgr2=0x7f07f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:49.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.718+0000 7f07fec3d700 1 -- 172.21.15.173:0/235002910 shutdown_connections 2024-02-19T21:57:49.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.718+0000 7f07fec3d700 1 -- 172.21.15.173:0/235002910 wait complete. 2024-02-19T21:57:49.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.719+0000 7f07fec3d700 1 Processor -- start 2024-02-19T21:57:49.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.719+0000 7f07fec3d700 1 -- start start 2024-02-19T21:57:49.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07fec3d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07f80f5410 0x7f07f81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:49.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07fec3d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8105bd0 con 0x7f07f80f5410 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07fc9d9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07f80f5410 0x7f07f81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07fc9d9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07f80f5410 0x7f07f81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:37584/0 (socket says 172.21.15.173:37584) 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07fc9d9700 1 -- 172.21.15.173:0/463130305 learned_addr learned my addr 172.21.15.173:0/463130305 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07fc9d9700 1 -- 172.21.15.173:0/463130305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec005040 con 0x7f07f80f5410 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07fc9d9700 1 --2- 172.21.15.173:0/463130305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07f80f5410 0x7f07f81056e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f07ec005010 tx=0x7f07ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07f57fa700 1 -- 172.21.15.173:0/463130305 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07ec009320 con 0x7f07f80f5410 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07f57fa700 1 -- 172.21.15.173:0/463130305 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07ec013380 con 0x7f07f80f5410 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07fec3d700 1 -- 172.21.15.173:0/463130305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f8105dd0 con 0x7f07f80f5410 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07f57fa700 1 -- 172.21.15.173:0/463130305 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07ec014e50 con 0x7f07f80f5410 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.720+0000 7f07fec3d700 1 -- 172.21.15.173:0/463130305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f8106260 con 0x7f07f80f5410 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.721+0000 7f07f57fa700 1 -- 172.21.15.173:0/463130305 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07ec014420 con 0x7f07f80f5410 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.721+0000 7f07fec3d700 1 -- 172.21.15.173:0/463130305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f8045ad0 con 0x7f07f80f5410 2024-02-19T21:57:49.764 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.722+0000 7f07f57fa700 1 --2- 172.21.15.173:0/463130305 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07e802ff10 0x7f07e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:49.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.722+0000 7f07f57fa700 1 -- 172.21.15.173:0/463130305 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07ec040e30 con 0x7f07f80f5410 2024-02-19T21:57:49.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.722+0000 7f07f7fff700 1 --2- 172.21.15.173:0/463130305 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07e802ff10 0x7f07e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:49.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.722+0000 7f07f7fff700 1 --2- 172.21.15.173:0/463130305 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07e802ff10 0x7f07e80323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f07e40096a0 tx=0x7f07e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:49.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.727+0000 7f07f57fa700 1 -- 172.21.15.173:0/463130305 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07ec0095f0 con 0x7f07f80f5410 2024-02-19T21:57:49.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.934+0000 7f07fec3d700 1 -- 172.21.15.173:0/463130305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07f8107110 con 0x7f07f80f5410 2024-02-19T21:57:49.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.935+0000 7f07f57fa700 1 -- 172.21.15.173:0/463130305 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f07ec01db70 con 0x7f07f80f5410 2024-02-19T21:57:49.935 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:49.935 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:49.936 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.937+0000 7f07fec3d700 1 -- 172.21.15.173:0/463130305 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07e802ff10 msgr2=0x7f07e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:49.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.937+0000 7f07fec3d700 1 --2- 172.21.15.173:0/463130305 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07e802ff10 0x7f07e80323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f07e40096a0 tx=0x7f07e4006b40).stop 2024-02-19T21:57:49.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.937+0000 7f07fec3d700 1 -- 172.21.15.173:0/463130305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07f80f5410 msgr2=0x7f07f81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:49.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.937+0000 7f07fec3d700 1 --2- 172.21.15.173:0/463130305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07f80f5410 0x7f07f81056e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f07ec005010 tx=0x7f07ec00b040).stop 2024-02-19T21:57:49.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.937+0000 7f07fec3d700 1 -- 172.21.15.173:0/463130305 shutdown_connections 2024-02-19T21:57:49.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.937+0000 7f07fec3d700 1 --2- 172.21.15.173:0/463130305 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f07e802ff10 0x7f07e80323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:49.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.937+0000 7f07fec3d700 1 --2- 172.21.15.173:0/463130305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07f80f5410 0x7f07f81056e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:49.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.937+0000 7f07fec3d700 1 -- 172.21.15.173:0/463130305 >> 172.21.15.173:0/463130305 conn(0x7f07f80f0b70 msgr2=0x7f07f80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:49.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.937+0000 7f07fec3d700 1 -- 172.21.15.173:0/463130305 shutdown_connections 2024-02-19T21:57:49.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:49.937+0000 7f07fec3d700 1 -- 172.21.15.173:0/463130305 wait complete. 2024-02-19T21:57:49.937 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:48.470363+0000 mgr.smithi143.bissbc (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:49.936273+0000 mon.smithi143 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.173:0/463130305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:51.681 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:51.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:52.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:50.470664+0000 mgr.smithi143.bissbc (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:53.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.688+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1415973060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f200f62a0 msgr2=0x7f7f200f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:53.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.688+0000 7f7f1f59e700 1 --2- 172.21.15.173:0/1415973060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f200f62a0 0x7f7f200f66c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f7f14009fb0 tx=0x7f7f14009b30).stop 2024-02-19T21:57:53.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.688+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1415973060 shutdown_connections 2024-02-19T21:57:53.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.688+0000 7f7f1f59e700 1 --2- 172.21.15.173:0/1415973060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f200f62a0 0x7f7f200f66c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:53.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.688+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1415973060 >> 172.21.15.173:0/1415973060 conn(0x7f7f200f1aa0 msgr2=0x7f7f200f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:53.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.688+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1415973060 shutdown_connections 2024-02-19T21:57:53.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.688+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1415973060 wait complete. 2024-02-19T21:57:53.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.688+0000 7f7f1f59e700 1 Processor -- start 2024-02-19T21:57:53.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.688+0000 7f7f1f59e700 1 -- start start 2024-02-19T21:57:53.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.688+0000 7f7f1f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f200f62a0 0x7f7f201087e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:53.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.688+0000 7f7f1f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f20108cd0 con 0x7f7f200f62a0 2024-02-19T21:57:53.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.689+0000 7f7f1e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f200f62a0 0x7f7f201087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:53.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.689+0000 7f7f1e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f200f62a0 0x7f7f201087e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:39892/0 (socket says 172.21.15.173:39892) 2024-02-19T21:57:53.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.689+0000 7f7f1e59c700 1 -- 172.21.15.173:0/1427505176 learned_addr learned my addr 172.21.15.173:0/1427505176 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:53.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.689+0000 7f7f1e59c700 1 -- 172.21.15.173:0/1427505176 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f14005040 con 0x7f7f200f62a0 2024-02-19T21:57:53.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.689+0000 7f7f1e59c700 1 --2- 172.21.15.173:0/1427505176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f200f62a0 0x7f7f201087e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f7f14009830 tx=0x7f7f1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:53.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.689+0000 7f7f0effd700 1 -- 172.21.15.173:0/1427505176 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f14009320 con 0x7f7f200f62a0 2024-02-19T21:57:53.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.689+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1427505176 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f20108ed0 con 0x7f7f200f62a0 2024-02-19T21:57:53.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.689+0000 7f7f0effd700 1 -- 172.21.15.173:0/1427505176 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f14013380 con 0x7f7f200f62a0 2024-02-19T21:57:53.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.689+0000 7f7f0effd700 1 -- 172.21.15.173:0/1427505176 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f14014420 con 0x7f7f200f62a0 2024-02-19T21:57:53.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.689+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1427505176 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f20109360 con 0x7f7f200f62a0 2024-02-19T21:57:53.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.690+0000 7f7f0effd700 1 -- 172.21.15.173:0/1427505176 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f1401e420 con 0x7f7f200f62a0 2024-02-19T21:57:53.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.690+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1427505176 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f20045ad0 con 0x7f7f200f62a0 2024-02-19T21:57:53.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.690+0000 7f7f0effd700 1 --2- 172.21.15.173:0/1427505176 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7f0802ff60 0x7f7f08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:53.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.690+0000 7f7f0effd700 1 -- 172.21.15.173:0/1427505176 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f14040e70 con 0x7f7f200f62a0 2024-02-19T21:57:53.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.691+0000 7f7f1dd9b700 1 --2- 172.21.15.173:0/1427505176 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7f0802ff60 0x7f7f08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:53.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.691+0000 7f7f1dd9b700 1 --2- 172.21.15.173:0/1427505176 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7f0802ff60 0x7f7f08032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7f100096a0 tx=0x7f7f10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:53.695 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.695+0000 7f7f0effd700 1 -- 172.21.15.173:0/1427505176 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f14014aa0 con 0x7f7f200f62a0 2024-02-19T21:57:53.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.901+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1427505176 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f200fbe30 con 0x7f7f200f62a0 2024-02-19T21:57:53.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.902+0000 7f7f0effd700 1 -- 172.21.15.173:0/1427505176 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7f14004490 con 0x7f7f200f62a0 2024-02-19T21:57:53.902 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:53.902 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:53.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.904+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1427505176 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7f0802ff60 msgr2=0x7f7f08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:53.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.904+0000 7f7f1f59e700 1 --2- 172.21.15.173:0/1427505176 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7f0802ff60 0x7f7f08032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7f100096a0 tx=0x7f7f10006b40).stop 2024-02-19T21:57:53.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.904+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1427505176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f200f62a0 msgr2=0x7f7f201087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:53.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.904+0000 7f7f1f59e700 1 --2- 172.21.15.173:0/1427505176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f200f62a0 0x7f7f201087e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f7f14009830 tx=0x7f7f1400b040).stop 2024-02-19T21:57:53.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.904+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1427505176 shutdown_connections 2024-02-19T21:57:53.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.904+0000 7f7f1f59e700 1 --2- 172.21.15.173:0/1427505176 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7f0802ff60 0x7f7f08032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:53.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.904+0000 7f7f1f59e700 1 --2- 172.21.15.173:0/1427505176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f200f62a0 0x7f7f201087e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:53.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.904+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1427505176 >> 172.21.15.173:0/1427505176 conn(0x7f7f200f1aa0 msgr2=0x7f7f200f2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:53.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.904+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1427505176 shutdown_connections 2024-02-19T21:57:53.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:53.904+0000 7f7f1f59e700 1 -- 172.21.15.173:0/1427505176 wait complete. 2024-02-19T21:57:53.905 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:54.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:52.471023+0000 mgr.smithi143.bissbc (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:54.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:53.903060+0000 mon.smithi143 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.173:0/1427505176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:55.660 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:55.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:54.471256+0000 mgr.smithi143.bissbc (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.639+0000 7f4dc9046700 1 -- 172.21.15.173:0/3941694365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc40f6510 msgr2=0x7f4dc40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.639+0000 7f4dc9046700 1 --2- 172.21.15.173:0/3941694365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc40f6510 0x7f4dc40f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4dac009fb0 tx=0x7f4dac009b30).stop 2024-02-19T21:57:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.639+0000 7f4dc9046700 1 -- 172.21.15.173:0/3941694365 shutdown_connections 2024-02-19T21:57:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.639+0000 7f4dc9046700 1 --2- 172.21.15.173:0/3941694365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc40f6510 0x7f4dc40f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:57.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.639+0000 7f4dc9046700 1 -- 172.21.15.173:0/3941694365 >> 172.21.15.173:0/3941694365 conn(0x7f4dc40f1c70 msgr2=0x7f4dc40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:57.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.639+0000 7f4dc9046700 1 -- 172.21.15.173:0/3941694365 shutdown_connections 2024-02-19T21:57:57.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.639+0000 7f4dc9046700 1 -- 172.21.15.173:0/3941694365 wait complete. 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.640+0000 7f4dc9046700 1 Processor -- start 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.640+0000 7f4dc9046700 1 -- start start 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.640+0000 7f4dc9046700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc40f6510 0x7f4dc406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.640+0000 7f4dc9046700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc4069e20 con 0x7f4dc40f6510 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.640+0000 7f4dc2d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc40f6510 0x7f4dc406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.640+0000 7f4dc2d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc40f6510 0x7f4dc406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:39900/0 (socket says 172.21.15.173:39900) 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.640+0000 7f4dc2d9d700 1 -- 172.21.15.173:0/2944059746 learned_addr learned my addr 172.21.15.173:0/2944059746 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.641+0000 7f4dc2d9d700 1 -- 172.21.15.173:0/2944059746 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac005040 con 0x7f4dc40f6510 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.641+0000 7f4dc2d9d700 1 --2- 172.21.15.173:0/2944059746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc40f6510 0x7f4dc406b9a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4dac005010 tx=0x7f4dac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.641+0000 7f4dbb7fe700 1 -- 172.21.15.173:0/2944059746 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dac009320 con 0x7f4dc40f6510 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.641+0000 7f4dbb7fe700 1 -- 172.21.15.173:0/2944059746 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4dac013380 con 0x7f4dc40f6510 2024-02-19T21:57:57.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.641+0000 7f4dc9046700 1 -- 172.21.15.173:0/2944059746 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc4069fc0 con 0x7f4dc40f6510 2024-02-19T21:57:57.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.641+0000 7f4dbb7fe700 1 -- 172.21.15.173:0/2944059746 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dac014e50 con 0x7f4dc40f6510 2024-02-19T21:57:57.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.641+0000 7f4dc9046700 1 -- 172.21.15.173:0/2944059746 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc406a450 con 0x7f4dc40f6510 2024-02-19T21:57:57.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.642+0000 7f4dbb7fe700 1 -- 172.21.15.173:0/2944059746 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4dac014420 con 0x7f4dc40f6510 2024-02-19T21:57:57.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.642+0000 7f4dbb7fe700 1 --2- 172.21.15.173:0/2944059746 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4db002ff10 0x7f4db00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:57.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.642+0000 7f4dc9046700 1 -- 172.21.15.173:0/2944059746 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc4059100 con 0x7f4dc40f6510 2024-02-19T21:57:57.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.642+0000 7f4dbb7fe700 1 -- 172.21.15.173:0/2944059746 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4dac040e30 con 0x7f4dc40f6510 2024-02-19T21:57:57.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.642+0000 7f4dc259c700 1 --2- 172.21.15.173:0/2944059746 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4db002ff10 0x7f4db00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:57.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.642+0000 7f4dc259c700 1 --2- 172.21.15.173:0/2944059746 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4db002ff10 0x7f4db00323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f4db40096a0 tx=0x7f4db4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:57.646 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.647+0000 7f4dbb7fe700 1 -- 172.21.15.173:0/2944059746 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4dac01db70 con 0x7f4dc40f6510 2024-02-19T21:57:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.853+0000 7f4dc9046700 1 -- 172.21.15.173:0/2944059746 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4dc40fc0a0 con 0x7f4dc40f6510 2024-02-19T21:57:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.854+0000 7f4dbb7fe700 1 -- 172.21.15.173:0/2944059746 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4dac023250 con 0x7f4dc40f6510 2024-02-19T21:57:57.854 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:57:57.854 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.855+0000 7f4dc9046700 1 -- 172.21.15.173:0/2944059746 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4db002ff10 msgr2=0x7f4db00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.855+0000 7f4dc9046700 1 --2- 172.21.15.173:0/2944059746 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4db002ff10 0x7f4db00323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f4db40096a0 tx=0x7f4db4006b40).stop 2024-02-19T21:57:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.855+0000 7f4dc9046700 1 -- 172.21.15.173:0/2944059746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc40f6510 msgr2=0x7f4dc406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.855+0000 7f4dc9046700 1 --2- 172.21.15.173:0/2944059746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc40f6510 0x7f4dc406b9a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4dac005010 tx=0x7f4dac00b040).stop 2024-02-19T21:57:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.855+0000 7f4dc9046700 1 -- 172.21.15.173:0/2944059746 shutdown_connections 2024-02-19T21:57:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.855+0000 7f4dc9046700 1 --2- 172.21.15.173:0/2944059746 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4db002ff10 0x7f4db00323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.855+0000 7f4dc9046700 1 --2- 172.21.15.173:0/2944059746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc40f6510 0x7f4dc406b9a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.855+0000 7f4dc9046700 1 -- 172.21.15.173:0/2944059746 >> 172.21.15.173:0/2944059746 conn(0x7f4dc40f1c70 msgr2=0x7f4dc40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.855+0000 7f4dc9046700 1 -- 172.21.15.173:0/2944059746 shutdown_connections 2024-02-19T21:57:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:57:57.855+0000 7f4dc9046700 1 -- 172.21.15.173:0/2944059746 wait complete. 2024-02-19T21:57:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:57:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:56.471517+0000 mgr.smithi143.bissbc (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:57:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:57:57.854904+0000 mon.smithi143 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.173:0/2944059746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:59.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:59.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:57:58.471824+0000 mgr.smithi143.bissbc (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.541+0000 7ff543225700 1 -- 172.21.15.173:0/3015713452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff53c0f5410 msgr2=0x7ff53c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.541+0000 7ff543225700 1 --2- 172.21.15.173:0/3015713452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff53c0f5410 0x7ff53c0f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff52c009fb0 tx=0x7ff52c009b30).stop 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.541+0000 7ff543225700 1 -- 172.21.15.173:0/3015713452 shutdown_connections 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.541+0000 7ff543225700 1 --2- 172.21.15.173:0/3015713452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff53c0f5410 0x7ff53c0f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.541+0000 7ff543225700 1 -- 172.21.15.173:0/3015713452 >> 172.21.15.173:0/3015713452 conn(0x7ff53c0f0b70 msgr2=0x7ff53c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.541+0000 7ff543225700 1 -- 172.21.15.173:0/3015713452 shutdown_connections 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.541+0000 7ff543225700 1 -- 172.21.15.173:0/3015713452 wait complete. 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.542+0000 7ff543225700 1 Processor -- start 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.542+0000 7ff543225700 1 -- start start 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.542+0000 7ff543225700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff53c0f5410 0x7ff53c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.542+0000 7ff543225700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c105bd0 con 0x7ff53c0f5410 2024-02-19T21:58:01.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.542+0000 7ff540fc1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff53c0f5410 0x7ff53c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:01.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.542+0000 7ff540fc1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff53c0f5410 0x7ff53c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:48444/0 (socket says 172.21.15.173:48444) 2024-02-19T21:58:01.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.542+0000 7ff540fc1700 1 -- 172.21.15.173:0/552158029 learned_addr learned my addr 172.21.15.173:0/552158029 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:01.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.542+0000 7ff540fc1700 1 -- 172.21.15.173:0/552158029 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c005040 con 0x7ff53c0f5410 2024-02-19T21:58:01.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.543+0000 7ff540fc1700 1 --2- 172.21.15.173:0/552158029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff53c0f5410 0x7ff53c1056e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7ff52c0089f0 tx=0x7ff52c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:01.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.543+0000 7ff5397fa700 1 -- 172.21.15.173:0/552158029 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff52c009320 con 0x7ff53c0f5410 2024-02-19T21:58:01.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.543+0000 7ff543225700 1 -- 172.21.15.173:0/552158029 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c105dd0 con 0x7ff53c0f5410 2024-02-19T21:58:01.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.543+0000 7ff5397fa700 1 -- 172.21.15.173:0/552158029 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff52c013380 con 0x7ff53c0f5410 2024-02-19T21:58:01.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.543+0000 7ff5397fa700 1 -- 172.21.15.173:0/552158029 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff52c01ed80 con 0x7ff53c0f5410 2024-02-19T21:58:01.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.543+0000 7ff543225700 1 -- 172.21.15.173:0/552158029 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c106260 con 0x7ff53c0f5410 2024-02-19T21:58:01.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.543+0000 7ff5397fa700 1 -- 172.21.15.173:0/552158029 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff52c014420 con 0x7ff53c0f5410 2024-02-19T21:58:01.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.544+0000 7ff5397fa700 1 --2- 172.21.15.173:0/552158029 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff52802ff60 0x7ff528032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:01.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.544+0000 7ff5397fa700 1 -- 172.21.15.173:0/552158029 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff52c041140 con 0x7ff53c0f5410 2024-02-19T21:58:01.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.544+0000 7ff543225700 1 -- 172.21.15.173:0/552158029 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53c059100 con 0x7ff53c0f5410 2024-02-19T21:58:01.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.544+0000 7ff53bfff700 1 --2- 172.21.15.173:0/552158029 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff52802ff60 0x7ff528032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:01.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.544+0000 7ff53bfff700 1 --2- 172.21.15.173:0/552158029 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff52802ff60 0x7ff528032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7ff5240096a0 tx=0x7ff524006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:01.549 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.549+0000 7ff5397fa700 1 -- 172.21.15.173:0/552158029 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff52c01c910 con 0x7ff53c0f5410 2024-02-19T21:58:01.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.757+0000 7ff543225700 1 -- 172.21.15.173:0/552158029 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff53c0fafa0 con 0x7ff53c0f5410 2024-02-19T21:58:01.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.757+0000 7ff5397fa700 1 -- 172.21.15.173:0/552158029 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff52c01c320 con 0x7ff53c0f5410 2024-02-19T21:58:01.757 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:01.758 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:01.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.759+0000 7ff543225700 1 -- 172.21.15.173:0/552158029 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff52802ff60 msgr2=0x7ff528032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:01.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.759+0000 7ff543225700 1 --2- 172.21.15.173:0/552158029 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff52802ff60 0x7ff528032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7ff5240096a0 tx=0x7ff524006b40).stop 2024-02-19T21:58:01.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.759+0000 7ff543225700 1 -- 172.21.15.173:0/552158029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff53c0f5410 msgr2=0x7ff53c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:01.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.759+0000 7ff543225700 1 --2- 172.21.15.173:0/552158029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff53c0f5410 0x7ff53c1056e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7ff52c0089f0 tx=0x7ff52c00b040).stop 2024-02-19T21:58:01.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.759+0000 7ff543225700 1 -- 172.21.15.173:0/552158029 shutdown_connections 2024-02-19T21:58:01.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.759+0000 7ff543225700 1 --2- 172.21.15.173:0/552158029 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff52802ff60 0x7ff528032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:01.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.759+0000 7ff543225700 1 --2- 172.21.15.173:0/552158029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff53c0f5410 0x7ff53c1056e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:01.759 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.759+0000 7ff543225700 1 -- 172.21.15.173:0/552158029 >> 172.21.15.173:0/552158029 conn(0x7ff53c0f0b70 msgr2=0x7ff53c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:01.760 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.759+0000 7ff543225700 1 -- 172.21.15.173:0/552158029 shutdown_connections 2024-02-19T21:58:01.760 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:01.759+0000 7ff543225700 1 -- 172.21.15.173:0/552158029 wait complete. 2024-02-19T21:58:01.760 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:00.472038+0000 mgr.smithi143.bissbc (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:01.758494+0000 mon.smithi143 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.173:0/552158029' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:03.463 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:03.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:02.472293+0000 mgr.smithi143.bissbc (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.439+0000 7fd116b64700 1 -- 172.21.15.173:0/4057206748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1100f6510 msgr2=0x7fd1100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.439+0000 7fd116b64700 1 --2- 172.21.15.173:0/4057206748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1100f6510 0x7fd1100f6930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fd104009fb0 tx=0x7fd104009b30).stop 2024-02-19T21:58:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.439+0000 7fd116b64700 1 -- 172.21.15.173:0/4057206748 shutdown_connections 2024-02-19T21:58:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.439+0000 7fd116b64700 1 --2- 172.21.15.173:0/4057206748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1100f6510 0x7fd1100f6930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.439+0000 7fd116b64700 1 -- 172.21.15.173:0/4057206748 >> 172.21.15.173:0/4057206748 conn(0x7fd1100f1c70 msgr2=0x7fd1100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.439+0000 7fd116b64700 1 -- 172.21.15.173:0/4057206748 shutdown_connections 2024-02-19T21:58:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.439+0000 7fd116b64700 1 -- 172.21.15.173:0/4057206748 wait complete. 2024-02-19T21:58:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.440+0000 7fd116b64700 1 Processor -- start 2024-02-19T21:58:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.440+0000 7fd116b64700 1 -- start start 2024-02-19T21:58:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.440+0000 7fd116b64700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1100f6510 0x7fd110106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.440+0000 7fd116b64700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd110106d00 con 0x7fd1100f6510 2024-02-19T21:58:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.441+0000 7fd114900700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1100f6510 0x7fd110106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.441+0000 7fd114900700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1100f6510 0x7fd110106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:48466/0 (socket says 172.21.15.173:48466) 2024-02-19T21:58:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.441+0000 7fd114900700 1 -- 172.21.15.173:0/2125064796 learned_addr learned my addr 172.21.15.173:0/2125064796 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.441+0000 7fd114900700 1 -- 172.21.15.173:0/2125064796 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd104005040 con 0x7fd1100f6510 2024-02-19T21:58:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.441+0000 7fd114900700 1 --2- 172.21.15.173:0/2125064796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1100f6510 0x7fd110106810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fd104005010 tx=0x7fd10400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.441+0000 7fd10d7fa700 1 -- 172.21.15.173:0/2125064796 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd104009320 con 0x7fd1100f6510 2024-02-19T21:58:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.441+0000 7fd116b64700 1 -- 172.21.15.173:0/2125064796 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd110106f00 con 0x7fd1100f6510 2024-02-19T21:58:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.441+0000 7fd10d7fa700 1 -- 172.21.15.173:0/2125064796 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd104013380 con 0x7fd1100f6510 2024-02-19T21:58:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.441+0000 7fd10d7fa700 1 -- 172.21.15.173:0/2125064796 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd104014e50 con 0x7fd1100f6510 2024-02-19T21:58:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.441+0000 7fd116b64700 1 -- 172.21.15.173:0/2125064796 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd110107390 con 0x7fd1100f6510 2024-02-19T21:58:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.442+0000 7fd10d7fa700 1 -- 172.21.15.173:0/2125064796 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1040263a0 con 0x7fd1100f6510 2024-02-19T21:58:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.442+0000 7fd116b64700 1 -- 172.21.15.173:0/2125064796 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd110045ad0 con 0x7fd1100f6510 2024-02-19T21:58:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.442+0000 7fd10d7fa700 1 --2- 172.21.15.173:0/2125064796 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd10002ff10 0x7fd1000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.442+0000 7fd10d7fa700 1 -- 172.21.15.173:0/2125064796 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd1040412a0 con 0x7fd1100f6510 2024-02-19T21:58:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.442+0000 7fd10ffff700 1 --2- 172.21.15.173:0/2125064796 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd10002ff10 0x7fd1000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.443+0000 7fd10ffff700 1 --2- 172.21.15.173:0/2125064796 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd10002ff10 0x7fd1000323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fd0fc0096a0 tx=0x7fd0fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.447+0000 7fd10d7fa700 1 -- 172.21.15.173:0/2125064796 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd104012510 con 0x7fd1100f6510 2024-02-19T21:58:05.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.651+0000 7fd116b64700 1 -- 172.21.15.173:0/2125064796 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd110108240 con 0x7fd1100f6510 2024-02-19T21:58:05.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.652+0000 7fd10d7fa700 1 -- 172.21.15.173:0/2125064796 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd104012330 con 0x7fd1100f6510 2024-02-19T21:58:05.652 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:05.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.654+0000 7fd116b64700 1 -- 172.21.15.173:0/2125064796 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd10002ff10 msgr2=0x7fd1000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:05.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.654+0000 7fd116b64700 1 --2- 172.21.15.173:0/2125064796 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd10002ff10 0x7fd1000323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fd0fc0096a0 tx=0x7fd0fc006b40).stop 2024-02-19T21:58:05.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.654+0000 7fd116b64700 1 -- 172.21.15.173:0/2125064796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1100f6510 msgr2=0x7fd110106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:05.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.654+0000 7fd116b64700 1 --2- 172.21.15.173:0/2125064796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1100f6510 0x7fd110106810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fd104005010 tx=0x7fd10400b040).stop 2024-02-19T21:58:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.654+0000 7fd116b64700 1 -- 172.21.15.173:0/2125064796 shutdown_connections 2024-02-19T21:58:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.654+0000 7fd116b64700 1 --2- 172.21.15.173:0/2125064796 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd10002ff10 0x7fd1000323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.654+0000 7fd116b64700 1 --2- 172.21.15.173:0/2125064796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1100f6510 0x7fd110106810 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.654+0000 7fd116b64700 1 -- 172.21.15.173:0/2125064796 >> 172.21.15.173:0/2125064796 conn(0x7fd1100f1c70 msgr2=0x7fd1100f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.654+0000 7fd116b64700 1 -- 172.21.15.173:0/2125064796 shutdown_connections 2024-02-19T21:58:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:05.654+0000 7fd116b64700 1 -- 172.21.15.173:0/2125064796 wait complete. 2024-02-19T21:58:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:04.472493+0000 mgr.smithi143.bissbc (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:05.653243+0000 mon.smithi143 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.173:0/2125064796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:07.225 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:07.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:06.472795+0000 mgr.smithi143.bissbc (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:09.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.273+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/2397244719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f40f5410 msgr2=0x7ff2f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:09.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.273+0000 7ff2f9fd7700 1 --2- 172.21.15.173:0/2397244719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f40f5410 0x7ff2f40f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7ff2dc009fb0 tx=0x7ff2dc009b30).stop 2024-02-19T21:58:09.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.273+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/2397244719 shutdown_connections 2024-02-19T21:58:09.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.273+0000 7ff2f9fd7700 1 --2- 172.21.15.173:0/2397244719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f40f5410 0x7ff2f40f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:09.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.273+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/2397244719 >> 172.21.15.173:0/2397244719 conn(0x7ff2f40f0b70 msgr2=0x7ff2f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:09.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.273+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/2397244719 shutdown_connections 2024-02-19T21:58:09.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.273+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/2397244719 wait complete. 2024-02-19T21:58:09.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.274+0000 7ff2f9fd7700 1 Processor -- start 2024-02-19T21:58:09.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.274+0000 7ff2f9fd7700 1 -- start start 2024-02-19T21:58:09.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.275+0000 7ff2f9fd7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f40f5410 0x7ff2f41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.275+0000 7ff2f9fd7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f4105bd0 con 0x7ff2f40f5410 2024-02-19T21:58:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.275+0000 7ff2f37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f40f5410 0x7ff2f41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.275+0000 7ff2f37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f40f5410 0x7ff2f41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:48490/0 (socket says 172.21.15.173:48490) 2024-02-19T21:58:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.275+0000 7ff2f37fe700 1 -- 172.21.15.173:0/4227704562 learned_addr learned my addr 172.21.15.173:0/4227704562 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.275+0000 7ff2f37fe700 1 -- 172.21.15.173:0/4227704562 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2dc005040 con 0x7ff2f40f5410 2024-02-19T21:58:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.275+0000 7ff2f37fe700 1 --2- 172.21.15.173:0/4227704562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f40f5410 0x7ff2f41056e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7ff2dc0089f0 tx=0x7ff2dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.275+0000 7ff2ebfff700 1 -- 172.21.15.173:0/4227704562 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2dc009320 con 0x7ff2f40f5410 2024-02-19T21:58:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.276+0000 7ff2ebfff700 1 -- 172.21.15.173:0/4227704562 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2dc013380 con 0x7ff2f40f5410 2024-02-19T21:58:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.276+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/4227704562 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f4105dd0 con 0x7ff2f40f5410 2024-02-19T21:58:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.276+0000 7ff2ebfff700 1 -- 172.21.15.173:0/4227704562 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2dc01ed80 con 0x7ff2f40f5410 2024-02-19T21:58:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.276+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/4227704562 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f4106260 con 0x7ff2f40f5410 2024-02-19T21:58:09.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.276+0000 7ff2ebfff700 1 -- 172.21.15.173:0/4227704562 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2dc01e420 con 0x7ff2f40f5410 2024-02-19T21:58:09.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.276+0000 7ff2ebfff700 1 --2- 172.21.15.173:0/4227704562 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff2e002ff60 0x7ff2e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:09.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.276+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/4227704562 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f4045ad0 con 0x7ff2f40f5410 2024-02-19T21:58:09.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.277+0000 7ff2ebfff700 1 -- 172.21.15.173:0/4227704562 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2dc040ef0 con 0x7ff2f40f5410 2024-02-19T21:58:09.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.277+0000 7ff2f2ffd700 1 --2- 172.21.15.173:0/4227704562 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff2e002ff60 0x7ff2e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:09.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.277+0000 7ff2f2ffd700 1 --2- 172.21.15.173:0/4227704562 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff2e002ff60 0x7ff2e0032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7ff2e40096a0 tx=0x7ff2e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:09.281 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.282+0000 7ff2ebfff700 1 -- 172.21.15.173:0/4227704562 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2dc0095f0 con 0x7ff2f40f5410 2024-02-19T21:58:09.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.488+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/4227704562 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2f4107110 con 0x7ff2f40f5410 2024-02-19T21:58:09.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.489+0000 7ff2ebfff700 1 -- 172.21.15.173:0/4227704562 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff2dc00e2a0 con 0x7ff2f40f5410 2024-02-19T21:58:09.489 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:09.489 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:09.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.491+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/4227704562 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff2e002ff60 msgr2=0x7ff2e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:09.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.491+0000 7ff2f9fd7700 1 --2- 172.21.15.173:0/4227704562 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff2e002ff60 0x7ff2e0032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7ff2e40096a0 tx=0x7ff2e4006b40).stop 2024-02-19T21:58:09.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.491+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/4227704562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f40f5410 msgr2=0x7ff2f41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:09.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.491+0000 7ff2f9fd7700 1 --2- 172.21.15.173:0/4227704562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f40f5410 0x7ff2f41056e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7ff2dc0089f0 tx=0x7ff2dc00b040).stop 2024-02-19T21:58:09.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.491+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/4227704562 shutdown_connections 2024-02-19T21:58:09.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.491+0000 7ff2f9fd7700 1 --2- 172.21.15.173:0/4227704562 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff2e002ff60 0x7ff2e0032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:09.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.491+0000 7ff2f9fd7700 1 --2- 172.21.15.173:0/4227704562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f40f5410 0x7ff2f41056e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:09.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.491+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/4227704562 >> 172.21.15.173:0/4227704562 conn(0x7ff2f40f0b70 msgr2=0x7ff2f40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:09.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.491+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/4227704562 shutdown_connections 2024-02-19T21:58:09.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:09.491+0000 7ff2f9fd7700 1 -- 172.21.15.173:0/4227704562 wait complete. 2024-02-19T21:58:09.491 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:08.473151+0000 mgr.smithi143.bissbc (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:09.490268+0000 mon.smithi143 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.173:0/4227704562' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:11.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:11.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:12.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:10.473486+0000 mgr.smithi143.bissbc (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:13.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.144+0000 7f1a89bb7700 1 -- 172.21.15.173:0/229530640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a840f5410 msgr2=0x7f1a840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:13.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.144+0000 7f1a89bb7700 1 --2- 172.21.15.173:0/229530640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a840f5410 0x7f1a840f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1a74009960 tx=0x7f1a74004d10).stop 2024-02-19T21:58:13.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.144+0000 7f1a89bb7700 1 -- 172.21.15.173:0/229530640 shutdown_connections 2024-02-19T21:58:13.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.144+0000 7f1a89bb7700 1 --2- 172.21.15.173:0/229530640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a840f5410 0x7f1a840f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:13.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.144+0000 7f1a89bb7700 1 -- 172.21.15.173:0/229530640 >> 172.21.15.173:0/229530640 conn(0x7f1a840f0b70 msgr2=0x7f1a840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:13.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.144+0000 7f1a89bb7700 1 -- 172.21.15.173:0/229530640 shutdown_connections 2024-02-19T21:58:13.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.144+0000 7f1a89bb7700 1 -- 172.21.15.173:0/229530640 wait complete. 2024-02-19T21:58:13.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.145+0000 7f1a89bb7700 1 Processor -- start 2024-02-19T21:58:13.146 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.145+0000 7f1a89bb7700 1 -- start start 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.145+0000 7f1a89bb7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a840f5410 0x7f1a841067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.145+0000 7f1a89bb7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a84106cd0 con 0x7f1a840f5410 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.145+0000 7f1a837fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a840f5410 0x7f1a841067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.145+0000 7f1a837fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a840f5410 0x7f1a841067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:54148/0 (socket says 172.21.15.173:54148) 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.145+0000 7f1a837fe700 1 -- 172.21.15.173:0/1984309721 learned_addr learned my addr 172.21.15.173:0/1984309721 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.146+0000 7f1a837fe700 1 -- 172.21.15.173:0/1984309721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a74009640 con 0x7f1a840f5410 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.146+0000 7f1a837fe700 1 --2- 172.21.15.173:0/1984309721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a840f5410 0x7f1a841067e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1a74004b70 tx=0x7f1a74004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.146+0000 7f1a6bfff700 1 -- 172.21.15.173:0/1984309721 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a74018070 con 0x7f1a840f5410 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.146+0000 7f1a6bfff700 1 -- 172.21.15.173:0/1984309721 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a7400b970 con 0x7f1a840f5410 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.146+0000 7f1a89bb7700 1 -- 172.21.15.173:0/1984309721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a84106ed0 con 0x7f1a840f5410 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.146+0000 7f1a6bfff700 1 -- 172.21.15.173:0/1984309721 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a7401c560 con 0x7f1a840f5410 2024-02-19T21:58:13.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.146+0000 7f1a89bb7700 1 -- 172.21.15.173:0/1984309721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a84107360 con 0x7f1a840f5410 2024-02-19T21:58:13.148 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.147+0000 7f1a6bfff700 1 -- 172.21.15.173:0/1984309721 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a74013630 con 0x7f1a840f5410 2024-02-19T21:58:13.148 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.147+0000 7f1a89bb7700 1 -- 172.21.15.173:0/1984309721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a84045ad0 con 0x7f1a840f5410 2024-02-19T21:58:13.148 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.147+0000 7f1a6bfff700 1 --2- 172.21.15.173:0/1984309721 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1a6c02ff60 0x7f1a6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:13.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.147+0000 7f1a6bfff700 1 -- 172.21.15.173:0/1984309721 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a74041310 con 0x7f1a840f5410 2024-02-19T21:58:13.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.147+0000 7f1a82ffd700 1 --2- 172.21.15.173:0/1984309721 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1a6c02ff60 0x7f1a6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:13.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.148+0000 7f1a82ffd700 1 --2- 172.21.15.173:0/1984309721 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1a6c02ff60 0x7f1a6c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f1a780096a0 tx=0x7f1a78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:13.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.152+0000 7f1a6bfff700 1 -- 172.21.15.173:0/1984309721 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a740139f0 con 0x7f1a840f5410 2024-02-19T21:58:13.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.356+0000 7f1a89bb7700 1 -- 172.21.15.173:0/1984309721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a840fafa0 con 0x7f1a840f5410 2024-02-19T21:58:13.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.356+0000 7f1a6bfff700 1 -- 172.21.15.173:0/1984309721 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1a74012380 con 0x7f1a840f5410 2024-02-19T21:58:13.356 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:13.356 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:13.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.358+0000 7f1a89bb7700 1 -- 172.21.15.173:0/1984309721 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1a6c02ff60 msgr2=0x7f1a6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:13.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.358+0000 7f1a89bb7700 1 --2- 172.21.15.173:0/1984309721 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1a6c02ff60 0x7f1a6c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f1a780096a0 tx=0x7f1a78006b40).stop 2024-02-19T21:58:13.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.358+0000 7f1a89bb7700 1 -- 172.21.15.173:0/1984309721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a840f5410 msgr2=0x7f1a841067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:13.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.358+0000 7f1a89bb7700 1 --2- 172.21.15.173:0/1984309721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a840f5410 0x7f1a841067e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1a74004b70 tx=0x7f1a74004d10).stop 2024-02-19T21:58:13.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.358+0000 7f1a89bb7700 1 -- 172.21.15.173:0/1984309721 shutdown_connections 2024-02-19T21:58:13.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.358+0000 7f1a89bb7700 1 --2- 172.21.15.173:0/1984309721 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1a6c02ff60 0x7f1a6c032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:13.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.358+0000 7f1a89bb7700 1 --2- 172.21.15.173:0/1984309721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a840f5410 0x7f1a841067e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:13.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.358+0000 7f1a89bb7700 1 -- 172.21.15.173:0/1984309721 >> 172.21.15.173:0/1984309721 conn(0x7f1a840f0b70 msgr2=0x7f1a8405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:13.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.358+0000 7f1a89bb7700 1 -- 172.21.15.173:0/1984309721 shutdown_connections 2024-02-19T21:58:13.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:13.358+0000 7f1a89bb7700 1 -- 172.21.15.173:0/1984309721 wait complete. 2024-02-19T21:58:13.359 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:13.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:13 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:13.357534+0000 mon.smithi143 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.173:0/1984309721' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:12.473896+0000 mgr.smithi143.bissbc (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:15.044 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:15.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:16.904 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.903+0000 7fa9b7fee700 1 -- 172.21.15.173:0/744965722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9b00f5410 msgr2=0x7fa9b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.903+0000 7fa9b7fee700 1 --2- 172.21.15.173:0/744965722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9b00f5410 0x7fa9b00f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa9a0009fb0 tx=0x7fa9a0009b30).stop 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.903+0000 7fa9b7fee700 1 -- 172.21.15.173:0/744965722 shutdown_connections 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.903+0000 7fa9b7fee700 1 --2- 172.21.15.173:0/744965722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9b00f5410 0x7fa9b00f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.903+0000 7fa9b7fee700 1 -- 172.21.15.173:0/744965722 >> 172.21.15.173:0/744965722 conn(0x7fa9b00f0b70 msgr2=0x7fa9b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.903+0000 7fa9b7fee700 1 -- 172.21.15.173:0/744965722 shutdown_connections 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.903+0000 7fa9b7fee700 1 -- 172.21.15.173:0/744965722 wait complete. 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.903+0000 7fa9b7fee700 1 Processor -- start 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9b7fee700 1 -- start start 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9b7fee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9b00f5410 0x7fa9b01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9b7fee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b0106cd0 con 0x7fa9b00f5410 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9b5d8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9b00f5410 0x7fa9b01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9b5d8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9b00f5410 0x7fa9b01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:54156/0 (socket says 172.21.15.173:54156) 2024-02-19T21:58:16.905 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9b5d8a700 1 -- 172.21.15.173:0/1755763400 learned_addr learned my addr 172.21.15.173:0/1755763400 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:16.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9b5d8a700 1 -- 172.21.15.173:0/1755763400 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a0005040 con 0x7fa9b00f5410 2024-02-19T21:58:16.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9b5d8a700 1 --2- 172.21.15.173:0/1755763400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9b00f5410 0x7fa9b01067e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fa9a0005010 tx=0x7fa9a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:16.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9a67fc700 1 -- 172.21.15.173:0/1755763400 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9a0009320 con 0x7fa9b00f5410 2024-02-19T21:58:16.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9b7fee700 1 -- 172.21.15.173:0/1755763400 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b0106ed0 con 0x7fa9b00f5410 2024-02-19T21:58:16.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9a67fc700 1 -- 172.21.15.173:0/1755763400 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9a0013380 con 0x7fa9b00f5410 2024-02-19T21:58:16.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9a67fc700 1 -- 172.21.15.173:0/1755763400 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9a0014e50 con 0x7fa9b00f5410 2024-02-19T21:58:16.906 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.904+0000 7fa9b7fee700 1 -- 172.21.15.173:0/1755763400 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b0107360 con 0x7fa9b00f5410 2024-02-19T21:58:16.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.905+0000 7fa9a67fc700 1 -- 172.21.15.173:0/1755763400 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9a00263a0 con 0x7fa9b00f5410 2024-02-19T21:58:16.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.905+0000 7fa9a67fc700 1 --2- 172.21.15.173:0/1755763400 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa99c02ff10 0x7fa99c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:16.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.905+0000 7fa9a67fc700 1 -- 172.21.15.173:0/1755763400 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9a0040ed0 con 0x7fa9b00f5410 2024-02-19T21:58:16.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.905+0000 7fa9b7fee700 1 -- 172.21.15.173:0/1755763400 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b0045ad0 con 0x7fa9b00f5410 2024-02-19T21:58:16.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.906+0000 7fa9b5589700 1 --2- 172.21.15.173:0/1755763400 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa99c02ff10 0x7fa99c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:16.907 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.906+0000 7fa9b5589700 1 --2- 172.21.15.173:0/1755763400 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa99c02ff10 0x7fa99c0323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fa9ac0096a0 tx=0x7fa9ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:14.474126+0000 mgr.smithi143.bissbc (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:16.910 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:16.911+0000 7fa9a67fc700 1 -- 172.21.15.173:0/1755763400 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9a0012920 con 0x7fa9b00f5410 2024-02-19T21:58:17.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.116+0000 7fa9b7fee700 1 -- 172.21.15.173:0/1755763400 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9b00fb090 con 0x7fa9b00f5410 2024-02-19T21:58:17.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.117+0000 7fa9a67fc700 1 -- 172.21.15.173:0/1755763400 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa9a0012330 con 0x7fa9b00f5410 2024-02-19T21:58:17.117 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:17.117 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:17.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.118+0000 7fa9b7fee700 1 -- 172.21.15.173:0/1755763400 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa99c02ff10 msgr2=0x7fa99c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:17.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.118+0000 7fa9b7fee700 1 --2- 172.21.15.173:0/1755763400 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa99c02ff10 0x7fa99c0323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fa9ac0096a0 tx=0x7fa9ac006b40).stop 2024-02-19T21:58:17.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.118+0000 7fa9b7fee700 1 -- 172.21.15.173:0/1755763400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9b00f5410 msgr2=0x7fa9b01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:17.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.118+0000 7fa9b7fee700 1 --2- 172.21.15.173:0/1755763400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9b00f5410 0x7fa9b01067e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fa9a0005010 tx=0x7fa9a000b040).stop 2024-02-19T21:58:17.119 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.118+0000 7fa9b7fee700 1 -- 172.21.15.173:0/1755763400 shutdown_connections 2024-02-19T21:58:17.119 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.118+0000 7fa9b7fee700 1 --2- 172.21.15.173:0/1755763400 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa99c02ff10 0x7fa99c0323d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:17.119 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.118+0000 7fa9b7fee700 1 --2- 172.21.15.173:0/1755763400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9b00f5410 0x7fa9b01067e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:17.119 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.118+0000 7fa9b7fee700 1 -- 172.21.15.173:0/1755763400 >> 172.21.15.173:0/1755763400 conn(0x7fa9b00f0b70 msgr2=0x7fa9b005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:17.119 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.119+0000 7fa9b7fee700 1 -- 172.21.15.173:0/1755763400 shutdown_connections 2024-02-19T21:58:17.119 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:17.119+0000 7fa9b7fee700 1 -- 172.21.15.173:0/1755763400 wait complete. 2024-02-19T21:58:17.119 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:17.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:17 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:17.118111+0000 mon.smithi143 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.173:0/1755763400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:18.888 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:18.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:16.474415+0000 mgr.smithi143.bissbc (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:20.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.889+0000 7f0f34d8b700 1 -- 172.21.15.173:0/17207204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f300f6510 msgr2=0x7f0f300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:20.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.889+0000 7f0f34d8b700 1 --2- 172.21.15.173:0/17207204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f300f6510 0x7f0f300f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f0f20009fb0 tx=0x7f0f20009b30).stop 2024-02-19T21:58:20.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.889+0000 7f0f34d8b700 1 -- 172.21.15.173:0/17207204 shutdown_connections 2024-02-19T21:58:20.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.889+0000 7f0f34d8b700 1 --2- 172.21.15.173:0/17207204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f300f6510 0x7f0f300f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:20.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.889+0000 7f0f34d8b700 1 -- 172.21.15.173:0/17207204 >> 172.21.15.173:0/17207204 conn(0x7f0f300f1c70 msgr2=0x7f0f300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:20.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.889+0000 7f0f34d8b700 1 -- 172.21.15.173:0/17207204 shutdown_connections 2024-02-19T21:58:20.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.889+0000 7f0f34d8b700 1 -- 172.21.15.173:0/17207204 wait complete. 2024-02-19T21:58:20.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.889+0000 7f0f34d8b700 1 Processor -- start 2024-02-19T21:58:20.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.890+0000 7f0f34d8b700 1 -- start start 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.890+0000 7f0f34d8b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f300f6510 0x7f0f3006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.890+0000 7f0f34d8b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f30069e20 con 0x7f0f300f6510 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.890+0000 7f0f2e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f300f6510 0x7f0f3006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.890+0000 7f0f2e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f300f6510 0x7f0f3006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:54176/0 (socket says 172.21.15.173:54176) 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.890+0000 7f0f2e59c700 1 -- 172.21.15.173:0/2982562155 learned_addr learned my addr 172.21.15.173:0/2982562155 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.890+0000 7f0f2e59c700 1 -- 172.21.15.173:0/2982562155 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f20005040 con 0x7f0f300f6510 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.891+0000 7f0f2e59c700 1 --2- 172.21.15.173:0/2982562155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f300f6510 0x7f0f3006b9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f0f20005010 tx=0x7f0f2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.891+0000 7f0f1effd700 1 -- 172.21.15.173:0/2982562155 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f20009320 con 0x7f0f300f6510 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.891+0000 7f0f1effd700 1 -- 172.21.15.173:0/2982562155 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f20013380 con 0x7f0f300f6510 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.891+0000 7f0f34d8b700 1 -- 172.21.15.173:0/2982562155 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f30069fc0 con 0x7f0f300f6510 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.891+0000 7f0f1effd700 1 -- 172.21.15.173:0/2982562155 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f20014e50 con 0x7f0f300f6510 2024-02-19T21:58:20.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.891+0000 7f0f34d8b700 1 -- 172.21.15.173:0/2982562155 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3006a450 con 0x7f0f300f6510 2024-02-19T21:58:20.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.892+0000 7f0f1effd700 1 -- 172.21.15.173:0/2982562155 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f200095f0 con 0x7f0f300f6510 2024-02-19T21:58:20.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.892+0000 7f0f34d8b700 1 -- 172.21.15.173:0/2982562155 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f30059100 con 0x7f0f300f6510 2024-02-19T21:58:20.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.892+0000 7f0f1effd700 1 --2- 172.21.15.173:0/2982562155 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0f1802ff10 0x7f0f180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:20.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.892+0000 7f0f1effd700 1 -- 172.21.15.173:0/2982562155 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f20040fa0 con 0x7f0f300f6510 2024-02-19T21:58:20.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.892+0000 7f0f2dd9b700 1 --2- 172.21.15.173:0/2982562155 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0f1802ff10 0x7f0f180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:20.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.892+0000 7f0f2dd9b700 1 --2- 172.21.15.173:0/2982562155 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0f1802ff10 0x7f0f180323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0f240096a0 tx=0x7f0f24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:20.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:20.897+0000 7f0f1effd700 1 -- 172.21.15.173:0/2982562155 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f20014420 con 0x7f0f300f6510 2024-02-19T21:58:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:18.474670+0000 mgr.smithi143.bissbc (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:21.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.103+0000 7f0f34d8b700 1 -- 172.21.15.173:0/2982562155 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f300fc0a0 con 0x7f0f300f6510 2024-02-19T21:58:21.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.104+0000 7f0f1effd700 1 -- 172.21.15.173:0/2982562155 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0f20010230 con 0x7f0f300f6510 2024-02-19T21:58:21.104 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:21.104 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:21.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.106+0000 7f0f34d8b700 1 -- 172.21.15.173:0/2982562155 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0f1802ff10 msgr2=0x7f0f180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:21.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.106+0000 7f0f34d8b700 1 --2- 172.21.15.173:0/2982562155 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0f1802ff10 0x7f0f180323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0f240096a0 tx=0x7f0f24006b40).stop 2024-02-19T21:58:21.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.106+0000 7f0f34d8b700 1 -- 172.21.15.173:0/2982562155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f300f6510 msgr2=0x7f0f3006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:21.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.106+0000 7f0f34d8b700 1 --2- 172.21.15.173:0/2982562155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f300f6510 0x7f0f3006b9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f0f20005010 tx=0x7f0f2000b040).stop 2024-02-19T21:58:21.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.106+0000 7f0f34d8b700 1 -- 172.21.15.173:0/2982562155 shutdown_connections 2024-02-19T21:58:21.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.106+0000 7f0f34d8b700 1 --2- 172.21.15.173:0/2982562155 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0f1802ff10 0x7f0f180323d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:21.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.106+0000 7f0f34d8b700 1 --2- 172.21.15.173:0/2982562155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f300f6510 0x7f0f3006b9a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:21.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.106+0000 7f0f34d8b700 1 -- 172.21.15.173:0/2982562155 >> 172.21.15.173:0/2982562155 conn(0x7f0f300f1c70 msgr2=0x7f0f300f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:21.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.106+0000 7f0f34d8b700 1 -- 172.21.15.173:0/2982562155 shutdown_connections 2024-02-19T21:58:21.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:21.106+0000 7f0f34d8b700 1 -- 172.21.15.173:0/2982562155 wait complete. 2024-02-19T21:58:21.107 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:20.474932+0000 mgr.smithi143.bissbc (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:20.564163+0000 mon.smithi143 (mon.0) 176 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:58:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:20.564391+0000 mon.smithi143 (mon.0) 177 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T21:58:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:21.104932+0000 mon.smithi143 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.173:0/2982562155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:22.738 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:22.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:24.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.768+0000 7faf8c5b2700 1 -- 172.21.15.173:0/2862912941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf840f6510 msgr2=0x7faf840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:24.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.768+0000 7faf8c5b2700 1 --2- 172.21.15.173:0/2862912941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf840f6510 0x7faf840f6930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7faf74009fb0 tx=0x7faf74009b30).stop 2024-02-19T21:58:24.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.768+0000 7faf8c5b2700 1 -- 172.21.15.173:0/2862912941 shutdown_connections 2024-02-19T21:58:24.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.768+0000 7faf8c5b2700 1 --2- 172.21.15.173:0/2862912941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf840f6510 0x7faf840f6930 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:24.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.768+0000 7faf8c5b2700 1 -- 172.21.15.173:0/2862912941 >> 172.21.15.173:0/2862912941 conn(0x7faf840f1c70 msgr2=0x7faf840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:24.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.768+0000 7faf8c5b2700 1 -- 172.21.15.173:0/2862912941 shutdown_connections 2024-02-19T21:58:24.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.768+0000 7faf8c5b2700 1 -- 172.21.15.173:0/2862912941 wait complete. 2024-02-19T21:58:24.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.769+0000 7faf8c5b2700 1 Processor -- start 2024-02-19T21:58:24.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.769+0000 7faf8c5b2700 1 -- start start 2024-02-19T21:58:24.769 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.769+0000 7faf8c5b2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf840f6510 0x7faf8406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:24.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.769+0000 7faf8c5b2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf84069e20 con 0x7faf840f6510 2024-02-19T21:58:24.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.769+0000 7faf8a34e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf840f6510 0x7faf8406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:24.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.769+0000 7faf8a34e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf840f6510 0x7faf8406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:42168/0 (socket says 172.21.15.173:42168) 2024-02-19T21:58:24.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.769+0000 7faf8a34e700 1 -- 172.21.15.173:0/1909062743 learned_addr learned my addr 172.21.15.173:0/1909062743 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:24.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.769+0000 7faf8a34e700 1 -- 172.21.15.173:0/1909062743 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf74005040 con 0x7faf840f6510 2024-02-19T21:58:24.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.770+0000 7faf8a34e700 1 --2- 172.21.15.173:0/1909062743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf840f6510 0x7faf8406b9a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7faf74005010 tx=0x7faf7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:24.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.770+0000 7faf7affd700 1 -- 172.21.15.173:0/1909062743 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf74009320 con 0x7faf840f6510 2024-02-19T21:58:24.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.770+0000 7faf8c5b2700 1 -- 172.21.15.173:0/1909062743 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf84069fc0 con 0x7faf840f6510 2024-02-19T21:58:24.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.770+0000 7faf7affd700 1 -- 172.21.15.173:0/1909062743 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf74013380 con 0x7faf840f6510 2024-02-19T21:58:24.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.770+0000 7faf7affd700 1 -- 172.21.15.173:0/1909062743 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf74014e50 con 0x7faf840f6510 2024-02-19T21:58:24.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.770+0000 7faf8c5b2700 1 -- 172.21.15.173:0/1909062743 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8406a450 con 0x7faf840f6510 2024-02-19T21:58:24.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.771+0000 7faf7affd700 1 -- 172.21.15.173:0/1909062743 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf740095f0 con 0x7faf840f6510 2024-02-19T21:58:24.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.771+0000 7faf8c5b2700 1 -- 172.21.15.173:0/1909062743 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf84059100 con 0x7faf840f6510 2024-02-19T21:58:24.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.771+0000 7faf7affd700 1 --2- 172.21.15.173:0/1909062743 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf7002ff10 0x7faf700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:24.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.771+0000 7faf7affd700 1 -- 172.21.15.173:0/1909062743 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf74040e90 con 0x7faf840f6510 2024-02-19T21:58:24.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.771+0000 7faf89b4d700 1 --2- 172.21.15.173:0/1909062743 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf7002ff10 0x7faf700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:24.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.771+0000 7faf89b4d700 1 --2- 172.21.15.173:0/1909062743 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf7002ff10 0x7faf700323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7faf800096a0 tx=0x7faf80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:24.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.776+0000 7faf7affd700 1 -- 172.21.15.173:0/1909062743 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf74014420 con 0x7faf840f6510 2024-02-19T21:58:24.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:22.475185+0000 mgr.smithi143.bissbc (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:24.982 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.982+0000 7faf8c5b2700 1 -- 172.21.15.173:0/1909062743 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf840fc0a0 con 0x7faf840f6510 2024-02-19T21:58:24.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.983+0000 7faf7affd700 1 -- 172.21.15.173:0/1909062743 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faf74010230 con 0x7faf840f6510 2024-02-19T21:58:24.983 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:24.983 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:24.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.984+0000 7faf8c5b2700 1 -- 172.21.15.173:0/1909062743 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf7002ff10 msgr2=0x7faf700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:24.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.984+0000 7faf8c5b2700 1 --2- 172.21.15.173:0/1909062743 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf7002ff10 0x7faf700323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7faf800096a0 tx=0x7faf80006b40).stop 2024-02-19T21:58:24.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.985+0000 7faf8c5b2700 1 -- 172.21.15.173:0/1909062743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf840f6510 msgr2=0x7faf8406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:24.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.985+0000 7faf8c5b2700 1 --2- 172.21.15.173:0/1909062743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf840f6510 0x7faf8406b9a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7faf74005010 tx=0x7faf7400b040).stop 2024-02-19T21:58:24.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.985+0000 7faf8c5b2700 1 -- 172.21.15.173:0/1909062743 shutdown_connections 2024-02-19T21:58:24.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.985+0000 7faf8c5b2700 1 --2- 172.21.15.173:0/1909062743 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf7002ff10 0x7faf700323d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:24.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.985+0000 7faf8c5b2700 1 --2- 172.21.15.173:0/1909062743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf840f6510 0x7faf8406b9a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:24.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.985+0000 7faf8c5b2700 1 -- 172.21.15.173:0/1909062743 >> 172.21.15.173:0/1909062743 conn(0x7faf840f1c70 msgr2=0x7faf840f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:24.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.985+0000 7faf8c5b2700 1 -- 172.21.15.173:0/1909062743 shutdown_connections 2024-02-19T21:58:24.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:24.985+0000 7faf8c5b2700 1 -- 172.21.15.173:0/1909062743 wait complete. 2024-02-19T21:58:24.985 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:25.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:25 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:24.984094+0000 mon.smithi143 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.173:0/1909062743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:26.707 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:26.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:24.475578+0000 mgr.smithi143.bissbc (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:28.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.678+0000 7f26e9c6a700 1 -- 172.21.15.173:0/642749001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e40f5410 msgr2=0x7f26e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:28.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.678+0000 7f26e9c6a700 1 --2- 172.21.15.173:0/642749001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e40f5410 0x7f26e40f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f26d8009fb0 tx=0x7f26d8009b30).stop 2024-02-19T21:58:28.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.678+0000 7f26e9c6a700 1 -- 172.21.15.173:0/642749001 shutdown_connections 2024-02-19T21:58:28.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.678+0000 7f26e9c6a700 1 --2- 172.21.15.173:0/642749001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e40f5410 0x7f26e40f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:28.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.678+0000 7f26e9c6a700 1 -- 172.21.15.173:0/642749001 >> 172.21.15.173:0/642749001 conn(0x7f26e40f0b70 msgr2=0x7f26e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:28.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.678+0000 7f26e9c6a700 1 -- 172.21.15.173:0/642749001 shutdown_connections 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.678+0000 7f26e9c6a700 1 -- 172.21.15.173:0/642749001 wait complete. 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.679+0000 7f26e9c6a700 1 Processor -- start 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.679+0000 7f26e9c6a700 1 -- start start 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.679+0000 7f26e9c6a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e40f5410 0x7f26e41094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.679+0000 7f26e9c6a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4060240 con 0x7f26e40f5410 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.679+0000 7f26e37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e40f5410 0x7f26e41094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.679+0000 7f26e37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e40f5410 0x7f26e41094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:42184/0 (socket says 172.21.15.173:42184) 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.679+0000 7f26e37fe700 1 -- 172.21.15.173:0/3039973918 learned_addr learned my addr 172.21.15.173:0/3039973918 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.679+0000 7f26e37fe700 1 -- 172.21.15.173:0/3039973918 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d8005040 con 0x7f26e40f5410 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.680+0000 7f26e37fe700 1 --2- 172.21.15.173:0/3039973918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e40f5410 0x7f26e41094e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f26d8005010 tx=0x7f26d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.680+0000 7f26cbfff700 1 -- 172.21.15.173:0/3039973918 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26d8009320 con 0x7f26e40f5410 2024-02-19T21:58:28.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.680+0000 7f26e9c6a700 1 -- 172.21.15.173:0/3039973918 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e4060440 con 0x7f26e40f5410 2024-02-19T21:58:28.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.680+0000 7f26cbfff700 1 -- 172.21.15.173:0/3039973918 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26d8013380 con 0x7f26e40f5410 2024-02-19T21:58:28.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.680+0000 7f26cbfff700 1 -- 172.21.15.173:0/3039973918 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26d8014e50 con 0x7f26e40f5410 2024-02-19T21:58:28.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.680+0000 7f26e9c6a700 1 -- 172.21.15.173:0/3039973918 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e4109b90 con 0x7f26e40f5410 2024-02-19T21:58:28.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.681+0000 7f26cbfff700 1 -- 172.21.15.173:0/3039973918 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26d80095f0 con 0x7f26e40f5410 2024-02-19T21:58:28.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.681+0000 7f26e9c6a700 1 -- 172.21.15.173:0/3039973918 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e4045ad0 con 0x7f26e40f5410 2024-02-19T21:58:28.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.681+0000 7f26cbfff700 1 --2- 172.21.15.173:0/3039973918 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f26cc02ff10 0x7f26cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:28.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.681+0000 7f26cbfff700 1 -- 172.21.15.173:0/3039973918 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26d8040e90 con 0x7f26e40f5410 2024-02-19T21:58:28.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.681+0000 7f26e2ffd700 1 --2- 172.21.15.173:0/3039973918 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f26cc02ff10 0x7f26cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:28.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.681+0000 7f26e2ffd700 1 --2- 172.21.15.173:0/3039973918 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f26cc02ff10 0x7f26cc0323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f26d40096a0 tx=0x7f26d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:28.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.686+0000 7f26cbfff700 1 -- 172.21.15.173:0/3039973918 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26d8014420 con 0x7f26e40f5410 2024-02-19T21:58:28.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.892+0000 7f26e9c6a700 1 -- 172.21.15.173:0/3039973918 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26e410a3b0 con 0x7f26e40f5410 2024-02-19T21:58:28.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.893+0000 7f26cbfff700 1 -- 172.21.15.173:0/3039973918 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f26d8010230 con 0x7f26e40f5410 2024-02-19T21:58:28.892 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:28.892 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:28.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.894+0000 7f26e9c6a700 1 -- 172.21.15.173:0/3039973918 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f26cc02ff10 msgr2=0x7f26cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:28.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.894+0000 7f26e9c6a700 1 --2- 172.21.15.173:0/3039973918 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f26cc02ff10 0x7f26cc0323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f26d40096a0 tx=0x7f26d4006b40).stop 2024-02-19T21:58:28.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.894+0000 7f26e9c6a700 1 -- 172.21.15.173:0/3039973918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e40f5410 msgr2=0x7f26e41094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:28.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.894+0000 7f26e9c6a700 1 --2- 172.21.15.173:0/3039973918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e40f5410 0x7f26e41094e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f26d8005010 tx=0x7f26d800b040).stop 2024-02-19T21:58:28.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.894+0000 7f26e9c6a700 1 -- 172.21.15.173:0/3039973918 shutdown_connections 2024-02-19T21:58:28.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.894+0000 7f26e9c6a700 1 --2- 172.21.15.173:0/3039973918 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f26cc02ff10 0x7f26cc0323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:28.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.894+0000 7f26e9c6a700 1 --2- 172.21.15.173:0/3039973918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e40f5410 0x7f26e41094e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:28.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.894+0000 7f26e9c6a700 1 -- 172.21.15.173:0/3039973918 >> 172.21.15.173:0/3039973918 conn(0x7f26e40f0b70 msgr2=0x7f26e40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:28.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.894+0000 7f26e9c6a700 1 -- 172.21.15.173:0/3039973918 shutdown_connections 2024-02-19T21:58:28.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:28.894+0000 7f26e9c6a700 1 -- 172.21.15.173:0/3039973918 wait complete. 2024-02-19T21:58:28.895 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:26.475847+0000 mgr.smithi143.bissbc (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:29.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:28.893835+0000 mon.smithi143 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.173:0/3039973918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:30.570 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:30.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:28.476210+0000 mgr.smithi143.bissbc (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:32.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.550+0000 7f804c008700 1 -- 172.21.15.173:0/399119831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80440f6510 msgr2=0x7f80440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:32.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.550+0000 7f804c008700 1 --2- 172.21.15.173:0/399119831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80440f6510 0x7f80440f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f8034009fb0 tx=0x7f8034009b30).stop 2024-02-19T21:58:32.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.550+0000 7f804c008700 1 -- 172.21.15.173:0/399119831 shutdown_connections 2024-02-19T21:58:32.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.550+0000 7f804c008700 1 --2- 172.21.15.173:0/399119831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80440f6510 0x7f80440f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:32.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.550+0000 7f804c008700 1 -- 172.21.15.173:0/399119831 >> 172.21.15.173:0/399119831 conn(0x7f80440f1c70 msgr2=0x7f80440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:32.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.550+0000 7f804c008700 1 -- 172.21.15.173:0/399119831 shutdown_connections 2024-02-19T21:58:32.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.550+0000 7f804c008700 1 -- 172.21.15.173:0/399119831 wait complete. 2024-02-19T21:58:32.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.550+0000 7f804c008700 1 Processor -- start 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.550+0000 7f804c008700 1 -- start start 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.550+0000 7f804c008700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80440f6510 0x7f804406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.550+0000 7f804c008700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8044069e20 con 0x7f80440f6510 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.551+0000 7f8049da4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80440f6510 0x7f804406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.551+0000 7f8049da4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80440f6510 0x7f804406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:50010/0 (socket says 172.21.15.173:50010) 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.551+0000 7f8049da4700 1 -- 172.21.15.173:0/2021059120 learned_addr learned my addr 172.21.15.173:0/2021059120 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.551+0000 7f8049da4700 1 -- 172.21.15.173:0/2021059120 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8034005040 con 0x7f80440f6510 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.551+0000 7f8049da4700 1 --2- 172.21.15.173:0/2021059120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80440f6510 0x7f804406b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f8034005010 tx=0x7f803400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.551+0000 7f803a7fc700 1 -- 172.21.15.173:0/2021059120 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8034009320 con 0x7f80440f6510 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.551+0000 7f803a7fc700 1 -- 172.21.15.173:0/2021059120 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8034013380 con 0x7f80440f6510 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.551+0000 7f804c008700 1 -- 172.21.15.173:0/2021059120 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8044069fc0 con 0x7f80440f6510 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.551+0000 7f803a7fc700 1 -- 172.21.15.173:0/2021059120 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8034014e50 con 0x7f80440f6510 2024-02-19T21:58:32.553 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.551+0000 7f804c008700 1 -- 172.21.15.173:0/2021059120 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f804406a450 con 0x7f80440f6510 2024-02-19T21:58:32.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.552+0000 7f803a7fc700 1 -- 172.21.15.173:0/2021059120 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80340095f0 con 0x7f80440f6510 2024-02-19T21:58:32.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.552+0000 7f804c008700 1 -- 172.21.15.173:0/2021059120 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8044059100 con 0x7f80440f6510 2024-02-19T21:58:32.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.552+0000 7f803a7fc700 1 --2- 172.21.15.173:0/2021059120 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f803002ff10 0x7f80300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:32.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.552+0000 7f803a7fc700 1 -- 172.21.15.173:0/2021059120 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8034040e90 con 0x7f80440f6510 2024-02-19T21:58:32.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.552+0000 7f80495a3700 1 --2- 172.21.15.173:0/2021059120 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f803002ff10 0x7f80300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:32.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.553+0000 7f80495a3700 1 --2- 172.21.15.173:0/2021059120 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f803002ff10 0x7f80300323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f80400096a0 tx=0x7f8040006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:32.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.558+0000 7f803a7fc700 1 -- 172.21.15.173:0/2021059120 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8034014420 con 0x7f80440f6510 2024-02-19T21:58:32.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.764+0000 7f804c008700 1 -- 172.21.15.173:0/2021059120 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80440fc0a0 con 0x7f80440f6510 2024-02-19T21:58:32.766 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.766+0000 7f803a7fc700 1 -- 172.21.15.173:0/2021059120 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8034010230 con 0x7f80440f6510 2024-02-19T21:58:32.766 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:32.766 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:32.767 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.766+0000 7f804c008700 1 -- 172.21.15.173:0/2021059120 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f803002ff10 msgr2=0x7f80300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:32.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.766+0000 7f804c008700 1 --2- 172.21.15.173:0/2021059120 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f803002ff10 0x7f80300323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f80400096a0 tx=0x7f8040006b40).stop 2024-02-19T21:58:32.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.767+0000 7f804c008700 1 -- 172.21.15.173:0/2021059120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80440f6510 msgr2=0x7f804406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:32.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.767+0000 7f804c008700 1 --2- 172.21.15.173:0/2021059120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80440f6510 0x7f804406b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f8034005010 tx=0x7f803400b040).stop 2024-02-19T21:58:32.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.767+0000 7f804c008700 1 -- 172.21.15.173:0/2021059120 shutdown_connections 2024-02-19T21:58:32.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.767+0000 7f804c008700 1 --2- 172.21.15.173:0/2021059120 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f803002ff10 0x7f80300323d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:32.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.767+0000 7f804c008700 1 --2- 172.21.15.173:0/2021059120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80440f6510 0x7f804406b9a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:32.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.767+0000 7f804c008700 1 -- 172.21.15.173:0/2021059120 >> 172.21.15.173:0/2021059120 conn(0x7f80440f1c70 msgr2=0x7f80440f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:32.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.767+0000 7f804c008700 1 -- 172.21.15.173:0/2021059120 shutdown_connections 2024-02-19T21:58:32.768 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:32.767+0000 7f804c008700 1 -- 172.21.15.173:0/2021059120 wait complete. 2024-02-19T21:58:32.769 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:32.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:30.476497+0000 mgr.smithi143.bissbc (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:33.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:33 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:32.767058+0000 mon.smithi143 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.173:0/2021059120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:34.450 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:34.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:32.476759+0000 mgr.smithi143.bissbc (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:36.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.325+0000 7f6dc97f5700 1 -- 172.21.15.173:0/2990994985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc40f6510 msgr2=0x7f6dc40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:36.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.325+0000 7f6dc97f5700 1 --2- 172.21.15.173:0/2990994985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc40f6510 0x7f6dc40f6930 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f6db8009fb0 tx=0x7f6db8009b30).stop 2024-02-19T21:58:36.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.325+0000 7f6dc97f5700 1 -- 172.21.15.173:0/2990994985 shutdown_connections 2024-02-19T21:58:36.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.325+0000 7f6dc97f5700 1 --2- 172.21.15.173:0/2990994985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc40f6510 0x7f6dc40f6930 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.325+0000 7f6dc97f5700 1 -- 172.21.15.173:0/2990994985 >> 172.21.15.173:0/2990994985 conn(0x7f6dc40f1c70 msgr2=0x7f6dc40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.325+0000 7f6dc97f5700 1 -- 172.21.15.173:0/2990994985 shutdown_connections 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.325+0000 7f6dc97f5700 1 -- 172.21.15.173:0/2990994985 wait complete. 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.326+0000 7f6dc97f5700 1 Processor -- start 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.326+0000 7f6dc97f5700 1 -- start start 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.326+0000 7f6dc97f5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc40f6510 0x7f6dc41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.326+0000 7f6dc97f5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc4106cd0 con 0x7f6dc40f6510 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.326+0000 7f6dc2ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc40f6510 0x7f6dc41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.326+0000 7f6dc2ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc40f6510 0x7f6dc41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:50020/0 (socket says 172.21.15.173:50020) 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.326+0000 7f6dc2ffd700 1 -- 172.21.15.173:0/3894339053 learned_addr learned my addr 172.21.15.173:0/3894339053 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.326+0000 7f6dc2ffd700 1 -- 172.21.15.173:0/3894339053 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db8005040 con 0x7f6dc40f6510 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.326+0000 7f6dc2ffd700 1 --2- 172.21.15.173:0/3894339053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc40f6510 0x7f6dc41067e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f6db80089f0 tx=0x7f6db800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:36.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.326+0000 7f6dab7fe700 1 -- 172.21.15.173:0/3894339053 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6db8009320 con 0x7f6dc40f6510 2024-02-19T21:58:36.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.327+0000 7f6dab7fe700 1 -- 172.21.15.173:0/3894339053 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6db8013380 con 0x7f6dc40f6510 2024-02-19T21:58:36.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.327+0000 7f6dab7fe700 1 -- 172.21.15.173:0/3894339053 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6db801ed80 con 0x7f6dc40f6510 2024-02-19T21:58:36.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.327+0000 7f6dc97f5700 1 -- 172.21.15.173:0/3894339053 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc4106ed0 con 0x7f6dc40f6510 2024-02-19T21:58:36.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.327+0000 7f6dc97f5700 1 -- 172.21.15.173:0/3894339053 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc4107360 con 0x7f6dc40f6510 2024-02-19T21:58:36.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.327+0000 7f6dab7fe700 1 -- 172.21.15.173:0/3894339053 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6db801e420 con 0x7f6dc40f6510 2024-02-19T21:58:36.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.327+0000 7f6dc97f5700 1 -- 172.21.15.173:0/3894339053 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc4045ad0 con 0x7f6dc40f6510 2024-02-19T21:58:36.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.328+0000 7f6dab7fe700 1 --2- 172.21.15.173:0/3894339053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6dac02ffb0 0x7f6dac032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:36.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.328+0000 7f6dab7fe700 1 -- 172.21.15.173:0/3894339053 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6db8040f30 con 0x7f6dc40f6510 2024-02-19T21:58:36.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.328+0000 7f6dc27fc700 1 --2- 172.21.15.173:0/3894339053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6dac02ffb0 0x7f6dac032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:36.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.328+0000 7f6dc27fc700 1 --2- 172.21.15.173:0/3894339053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6dac02ffb0 0x7f6dac032470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6db40096a0 tx=0x7f6db4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:36.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.332+0000 7f6dab7fe700 1 -- 172.21.15.173:0/3894339053 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6db80095f0 con 0x7f6dc40f6510 2024-02-19T21:58:36.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.537+0000 7f6dc97f5700 1 -- 172.21.15.173:0/3894339053 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6dc4108210 con 0x7f6dc40f6510 2024-02-19T21:58:36.538 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.538+0000 7f6dab7fe700 1 -- 172.21.15.173:0/3894339053 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6db801cb10 con 0x7f6dc40f6510 2024-02-19T21:58:36.538 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:36.538 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:36.539 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.540+0000 7f6dc97f5700 1 -- 172.21.15.173:0/3894339053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6dac02ffb0 msgr2=0x7f6dac032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:36.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.540+0000 7f6dc97f5700 1 --2- 172.21.15.173:0/3894339053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6dac02ffb0 0x7f6dac032470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6db40096a0 tx=0x7f6db4006b40).stop 2024-02-19T21:58:36.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.540+0000 7f6dc97f5700 1 -- 172.21.15.173:0/3894339053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc40f6510 msgr2=0x7f6dc41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:36.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.540+0000 7f6dc97f5700 1 --2- 172.21.15.173:0/3894339053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc40f6510 0x7f6dc41067e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f6db80089f0 tx=0x7f6db800b040).stop 2024-02-19T21:58:36.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.540+0000 7f6dc97f5700 1 -- 172.21.15.173:0/3894339053 shutdown_connections 2024-02-19T21:58:36.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.540+0000 7f6dc97f5700 1 --2- 172.21.15.173:0/3894339053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6dac02ffb0 0x7f6dac032470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:36.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.540+0000 7f6dc97f5700 1 --2- 172.21.15.173:0/3894339053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dc40f6510 0x7f6dc41067e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:36.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.540+0000 7f6dc97f5700 1 -- 172.21.15.173:0/3894339053 >> 172.21.15.173:0/3894339053 conn(0x7f6dc40f1c70 msgr2=0x7f6dc40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:36.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.540+0000 7f6dc97f5700 1 -- 172.21.15.173:0/3894339053 shutdown_connections 2024-02-19T21:58:36.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:36.540+0000 7f6dc97f5700 1 -- 172.21.15.173:0/3894339053 wait complete. 2024-02-19T21:58:36.540 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:34.476974+0000 mgr.smithi143.bissbc (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:37.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:37 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:36.539204+0000 mon.smithi143 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.173:0/3894339053' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:38.299 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:38.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:36.477220+0000 mgr.smithi143.bissbc (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:39.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:39 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:38.477551+0000 mgr.smithi143.bissbc (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:40.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.337+0000 7ff6f159d700 1 -- 172.21.15.173:0/282305286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6ec0f62a0 msgr2=0x7ff6ec0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:40.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.337+0000 7ff6f159d700 1 --2- 172.21.15.173:0/282305286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6ec0f62a0 0x7ff6ec0f66c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7ff6dc009fb0 tx=0x7ff6dc009b30).stop 2024-02-19T21:58:40.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.337+0000 7ff6f159d700 1 -- 172.21.15.173:0/282305286 shutdown_connections 2024-02-19T21:58:40.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.337+0000 7ff6f159d700 1 --2- 172.21.15.173:0/282305286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6ec0f62a0 0x7ff6ec0f66c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:40.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.337+0000 7ff6f159d700 1 -- 172.21.15.173:0/282305286 >> 172.21.15.173:0/282305286 conn(0x7ff6ec0f1aa0 msgr2=0x7ff6ec0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:40.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.337+0000 7ff6f159d700 1 -- 172.21.15.173:0/282305286 shutdown_connections 2024-02-19T21:58:40.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.337+0000 7ff6f159d700 1 -- 172.21.15.173:0/282305286 wait complete. 2024-02-19T21:58:40.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.338+0000 7ff6f159d700 1 Processor -- start 2024-02-19T21:58:40.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.338+0000 7ff6f159d700 1 -- start start 2024-02-19T21:58:40.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.338+0000 7ff6f159d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6ec0f62a0 0x7ff6ec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:40.338 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.338+0000 7ff6f159d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec069e20 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.338+0000 7ff6ebfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6ec0f62a0 0x7ff6ec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:40.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.338+0000 7ff6ebfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6ec0f62a0 0x7ff6ec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:50038/0 (socket says 172.21.15.173:50038) 2024-02-19T21:58:40.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.338+0000 7ff6ebfff700 1 -- 172.21.15.173:0/3294598967 learned_addr learned my addr 172.21.15.173:0/3294598967 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:40.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.339+0000 7ff6ebfff700 1 -- 172.21.15.173:0/3294598967 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc005040 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.339+0000 7ff6ebfff700 1 --2- 172.21.15.173:0/3294598967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6ec0f62a0 0x7ff6ec06b9a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7ff6dc009830 tx=0x7ff6dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:40.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.339+0000 7ff6e8ff9700 1 -- 172.21.15.173:0/3294598967 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6dc009320 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.339+0000 7ff6f159d700 1 -- 172.21.15.173:0/3294598967 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ec069fc0 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.339+0000 7ff6e8ff9700 1 -- 172.21.15.173:0/3294598967 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6dc013380 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.339+0000 7ff6e8ff9700 1 -- 172.21.15.173:0/3294598967 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6dc014420 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.339+0000 7ff6f159d700 1 -- 172.21.15.173:0/3294598967 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ec06a450 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.340+0000 7ff6e8ff9700 1 -- 172.21.15.173:0/3294598967 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6dc014580 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.340+0000 7ff6f159d700 1 -- 172.21.15.173:0/3294598967 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ec059100 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.340+0000 7ff6e8ff9700 1 --2- 172.21.15.173:0/3294598967 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff6d402ff60 0x7ff6d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:40.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.340+0000 7ff6e8ff9700 1 -- 172.21.15.173:0/3294598967 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6dc041100 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.340+0000 7ff6eb7fe700 1 --2- 172.21.15.173:0/3294598967 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff6d402ff60 0x7ff6d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:40.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.340+0000 7ff6eb7fe700 1 --2- 172.21.15.173:0/3294598967 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff6d402ff60 0x7ff6d4032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7ff6e00096a0 tx=0x7ff6e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:40.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.345+0000 7ff6e8ff9700 1 -- 172.21.15.173:0/3294598967 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6dc01c890 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.548 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.549+0000 7ff6f159d700 1 -- 172.21.15.173:0/3294598967 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6ec045ad0 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.549 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.549+0000 7ff6e8ff9700 1 -- 172.21.15.173:0/3294598967 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff6dc01c890 con 0x7ff6ec0f62a0 2024-02-19T21:58:40.549 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:40.549 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:40.550 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.551+0000 7ff6f159d700 1 -- 172.21.15.173:0/3294598967 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff6d402ff60 msgr2=0x7ff6d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:40.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.551+0000 7ff6f159d700 1 --2- 172.21.15.173:0/3294598967 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff6d402ff60 0x7ff6d4032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7ff6e00096a0 tx=0x7ff6e0006b40).stop 2024-02-19T21:58:40.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.551+0000 7ff6f159d700 1 -- 172.21.15.173:0/3294598967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6ec0f62a0 msgr2=0x7ff6ec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:40.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.551+0000 7ff6f159d700 1 --2- 172.21.15.173:0/3294598967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6ec0f62a0 0x7ff6ec06b9a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7ff6dc009830 tx=0x7ff6dc00b040).stop 2024-02-19T21:58:40.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.551+0000 7ff6f159d700 1 -- 172.21.15.173:0/3294598967 shutdown_connections 2024-02-19T21:58:40.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.551+0000 7ff6f159d700 1 --2- 172.21.15.173:0/3294598967 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff6d402ff60 0x7ff6d4032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:40.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.551+0000 7ff6f159d700 1 --2- 172.21.15.173:0/3294598967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6ec0f62a0 0x7ff6ec06b9a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:40.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.551+0000 7ff6f159d700 1 -- 172.21.15.173:0/3294598967 >> 172.21.15.173:0/3294598967 conn(0x7ff6ec0f1aa0 msgr2=0x7ff6ec0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:40.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.551+0000 7ff6f159d700 1 -- 172.21.15.173:0/3294598967 shutdown_connections 2024-02-19T21:58:40.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:40.551+0000 7ff6f159d700 1 -- 172.21.15.173:0/3294598967 wait complete. 2024-02-19T21:58:40.552 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:41.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:41 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:40.477899+0000 mgr.smithi143.bissbc (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:41.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:41 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:40.550413+0000 mon.smithi143 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.173:0/3294598967' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:42.141 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:42.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:44.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.079+0000 7f776f92b700 1 -- 172.21.15.173:0/2709975804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77680f5410 msgr2=0x7f77680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:44.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.079+0000 7f776f92b700 1 --2- 172.21.15.173:0/2709975804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77680f5410 0x7f77680f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f775c009fb0 tx=0x7f775c009b30).stop 2024-02-19T21:58:44.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.079+0000 7f776f92b700 1 -- 172.21.15.173:0/2709975804 shutdown_connections 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.079+0000 7f776f92b700 1 --2- 172.21.15.173:0/2709975804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77680f5410 0x7f77680f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.079+0000 7f776f92b700 1 -- 172.21.15.173:0/2709975804 >> 172.21.15.173:0/2709975804 conn(0x7f77680f0b70 msgr2=0x7f77680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.080+0000 7f776f92b700 1 -- 172.21.15.173:0/2709975804 shutdown_connections 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.080+0000 7f776f92b700 1 -- 172.21.15.173:0/2709975804 wait complete. 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.080+0000 7f776f92b700 1 Processor -- start 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.080+0000 7f776f92b700 1 -- start start 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.080+0000 7f776f92b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77680f5410 0x7f77680fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.080+0000 7f776f92b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77680fb960 con 0x7f77680f5410 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.081+0000 7f776d6c7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77680f5410 0x7f77680fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.081+0000 7f776d6c7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77680f5410 0x7f77680fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:44102/0 (socket says 172.21.15.173:44102) 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.081+0000 7f776d6c7700 1 -- 172.21.15.173:0/1764225465 learned_addr learned my addr 172.21.15.173:0/1764225465 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:44.082 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.081+0000 7f776d6c7700 1 -- 172.21.15.173:0/1764225465 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c005040 con 0x7f77680f5410 2024-02-19T21:58:44.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.081+0000 7f776d6c7700 1 --2- 172.21.15.173:0/1764225465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77680f5410 0x7f77680fd4e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f775c005010 tx=0x7f775c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:44.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.081+0000 7f7759ffb700 1 -- 172.21.15.173:0/1764225465 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f775c009320 con 0x7f77680f5410 2024-02-19T21:58:44.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.081+0000 7f776f92b700 1 -- 172.21.15.173:0/1764225465 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77680fbb00 con 0x7f77680f5410 2024-02-19T21:58:44.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.081+0000 7f7759ffb700 1 -- 172.21.15.173:0/1764225465 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f775c013380 con 0x7f77680f5410 2024-02-19T21:58:44.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.081+0000 7f7759ffb700 1 -- 172.21.15.173:0/1764225465 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f775c014e50 con 0x7f77680f5410 2024-02-19T21:58:44.083 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.081+0000 7f776f92b700 1 -- 172.21.15.173:0/1764225465 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77680fbf90 con 0x7f77680f5410 2024-02-19T21:58:44.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.082+0000 7f7759ffb700 1 -- 172.21.15.173:0/1764225465 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f775c0095f0 con 0x7f77680f5410 2024-02-19T21:58:44.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.082+0000 7f776f92b700 1 -- 172.21.15.173:0/1764225465 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7768059100 con 0x7f77680f5410 2024-02-19T21:58:44.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.082+0000 7f7759ffb700 1 --2- 172.21.15.173:0/1764225465 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f775402ff10 0x7f77540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:44.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.082+0000 7f7759ffb700 1 -- 172.21.15.173:0/1764225465 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f775c040f80 con 0x7f77680f5410 2024-02-19T21:58:44.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.083+0000 7f776cec6700 1 --2- 172.21.15.173:0/1764225465 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f775402ff10 0x7f77540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:44.084 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.083+0000 7f776cec6700 1 --2- 172.21.15.173:0/1764225465 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f775402ff10 0x7f77540323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f77640096a0 tx=0x7f7764006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:44.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.087+0000 7f7759ffb700 1 -- 172.21.15.173:0/1764225465 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f775c014420 con 0x7f77680f5410 2024-02-19T21:58:44.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.291+0000 7f776f92b700 1 -- 172.21.15.173:0/1764225465 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7768045ad0 con 0x7f77680f5410 2024-02-19T21:58:44.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.292+0000 7f7759ffb700 1 -- 172.21.15.173:0/1764225465 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f775c010230 con 0x7f77680f5410 2024-02-19T21:58:44.292 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:44.292 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:44.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.293+0000 7f776f92b700 1 -- 172.21.15.173:0/1764225465 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f775402ff10 msgr2=0x7f77540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:44.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.293+0000 7f776f92b700 1 --2- 172.21.15.173:0/1764225465 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f775402ff10 0x7f77540323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f77640096a0 tx=0x7f7764006b40).stop 2024-02-19T21:58:44.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.293+0000 7f776f92b700 1 -- 172.21.15.173:0/1764225465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77680f5410 msgr2=0x7f77680fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:44.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.293+0000 7f776f92b700 1 --2- 172.21.15.173:0/1764225465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77680f5410 0x7f77680fd4e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f775c005010 tx=0x7f775c00b040).stop 2024-02-19T21:58:44.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.293+0000 7f776f92b700 1 -- 172.21.15.173:0/1764225465 shutdown_connections 2024-02-19T21:58:44.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.293+0000 7f776f92b700 1 --2- 172.21.15.173:0/1764225465 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f775402ff10 0x7f77540323d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:44.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.293+0000 7f776f92b700 1 --2- 172.21.15.173:0/1764225465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77680f5410 0x7f77680fd4e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:44.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.293+0000 7f776f92b700 1 -- 172.21.15.173:0/1764225465 >> 172.21.15.173:0/1764225465 conn(0x7f77680f0b70 msgr2=0x7f776805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:44.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.293+0000 7f776f92b700 1 -- 172.21.15.173:0/1764225465 shutdown_connections 2024-02-19T21:58:44.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:44.294+0000 7f776f92b700 1 -- 172.21.15.173:0/1764225465 wait complete. 2024-02-19T21:58:44.294 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:42.478160+0000 mgr.smithi143.bissbc (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:44.293035+0000 mon.smithi143 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.173:0/1764225465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:45.897 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:45.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:44.478431+0000 mgr.smithi143.bissbc (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:47.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.845+0000 7f88a835b700 1 -- 172.21.15.173:0/3487454575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00f6510 msgr2=0x7f88a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:47.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.845+0000 7f88a835b700 1 --2- 172.21.15.173:0/3487454575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00f6510 0x7f88a00f6930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f8890009fb0 tx=0x7f8890009b30).stop 2024-02-19T21:58:47.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.845+0000 7f88a835b700 1 -- 172.21.15.173:0/3487454575 shutdown_connections 2024-02-19T21:58:47.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.845+0000 7f88a835b700 1 --2- 172.21.15.173:0/3487454575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00f6510 0x7f88a00f6930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:47.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.845+0000 7f88a835b700 1 -- 172.21.15.173:0/3487454575 >> 172.21.15.173:0/3487454575 conn(0x7f88a00f1c70 msgr2=0x7f88a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:47.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.845+0000 7f88a835b700 1 -- 172.21.15.173:0/3487454575 shutdown_connections 2024-02-19T21:58:47.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.845+0000 7f88a835b700 1 -- 172.21.15.173:0/3487454575 wait complete. 2024-02-19T21:58:47.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.845+0000 7f88a835b700 1 Processor -- start 2024-02-19T21:58:47.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.845+0000 7f88a835b700 1 -- start start 2024-02-19T21:58:47.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.846+0000 7f88a835b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00f6510 0x7f88a01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:47.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.846+0000 7f88a835b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a0106cd0 con 0x7f88a00f6510 2024-02-19T21:58:47.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.846+0000 7f88a60f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00f6510 0x7f88a01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:47.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.846+0000 7f88a60f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00f6510 0x7f88a01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:44134/0 (socket says 172.21.15.173:44134) 2024-02-19T21:58:47.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.846+0000 7f88a60f7700 1 -- 172.21.15.173:0/1141757249 learned_addr learned my addr 172.21.15.173:0/1141757249 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:47.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.846+0000 7f88a60f7700 1 -- 172.21.15.173:0/1141757249 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8890005040 con 0x7f88a00f6510 2024-02-19T21:58:47.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.846+0000 7f88a60f7700 1 --2- 172.21.15.173:0/1141757249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00f6510 0x7f88a01067e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f8890005010 tx=0x7f889000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:47.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.846+0000 7f8896ffd700 1 -- 172.21.15.173:0/1141757249 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8890009320 con 0x7f88a00f6510 2024-02-19T21:58:47.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.847+0000 7f8896ffd700 1 -- 172.21.15.173:0/1141757249 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8890013380 con 0x7f88a00f6510 2024-02-19T21:58:47.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.847+0000 7f8896ffd700 1 -- 172.21.15.173:0/1141757249 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8890014e50 con 0x7f88a00f6510 2024-02-19T21:58:47.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.847+0000 7f88a835b700 1 -- 172.21.15.173:0/1141757249 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a0106ed0 con 0x7f88a00f6510 2024-02-19T21:58:47.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.847+0000 7f88a835b700 1 -- 172.21.15.173:0/1141757249 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a0107360 con 0x7f88a00f6510 2024-02-19T21:58:47.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.847+0000 7f8896ffd700 1 -- 172.21.15.173:0/1141757249 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88900095f0 con 0x7f88a00f6510 2024-02-19T21:58:47.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.847+0000 7f88a835b700 1 -- 172.21.15.173:0/1141757249 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a0045ad0 con 0x7f88a00f6510 2024-02-19T21:58:47.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.847+0000 7f8896ffd700 1 --2- 172.21.15.173:0/1141757249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f888c02ff10 0x7f888c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:47.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.847+0000 7f8896ffd700 1 -- 172.21.15.173:0/1141757249 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8890040ee0 con 0x7f88a00f6510 2024-02-19T21:58:47.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.848+0000 7f88a58f6700 1 --2- 172.21.15.173:0/1141757249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f888c02ff10 0x7f888c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:47.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.848+0000 7f88a58f6700 1 --2- 172.21.15.173:0/1141757249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f888c02ff10 0x7f888c0323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f889c0096a0 tx=0x7f889c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:47.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:47.853+0000 7f8896ffd700 1 -- 172.21.15.173:0/1141757249 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8890014420 con 0x7f88a00f6510 2024-02-19T21:58:48.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.056+0000 7f88a835b700 1 -- 172.21.15.173:0/1141757249 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88a0108210 con 0x7f88a00f6510 2024-02-19T21:58:48.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.057+0000 7f8896ffd700 1 -- 172.21.15.173:0/1141757249 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f889001d980 con 0x7f88a00f6510 2024-02-19T21:58:48.057 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:48.057 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.058+0000 7f88a835b700 1 -- 172.21.15.173:0/1141757249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f888c02ff10 msgr2=0x7f888c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.058+0000 7f88a835b700 1 --2- 172.21.15.173:0/1141757249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f888c02ff10 0x7f888c0323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f889c0096a0 tx=0x7f889c006b40).stop 2024-02-19T21:58:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.058+0000 7f88a835b700 1 -- 172.21.15.173:0/1141757249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00f6510 msgr2=0x7f88a01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.058+0000 7f88a835b700 1 --2- 172.21.15.173:0/1141757249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00f6510 0x7f88a01067e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f8890005010 tx=0x7f889000b040).stop 2024-02-19T21:58:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.058+0000 7f88a835b700 1 -- 172.21.15.173:0/1141757249 shutdown_connections 2024-02-19T21:58:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.058+0000 7f88a835b700 1 --2- 172.21.15.173:0/1141757249 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f888c02ff10 0x7f888c0323d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.058+0000 7f88a835b700 1 --2- 172.21.15.173:0/1141757249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a00f6510 0x7f88a01067e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.058+0000 7f88a835b700 1 -- 172.21.15.173:0/1141757249 >> 172.21.15.173:0/1141757249 conn(0x7f88a00f1c70 msgr2=0x7f88a00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.058+0000 7f88a835b700 1 -- 172.21.15.173:0/1141757249 shutdown_connections 2024-02-19T21:58:48.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:48.058+0000 7f88a835b700 1 -- 172.21.15.173:0/1141757249 wait complete. 2024-02-19T21:58:48.059 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:46.478689+0000 mgr.smithi143.bissbc (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:48.057904+0000 mon.smithi143 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.173:0/1141757249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:49.795 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:49.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:48.479053+0000 mgr.smithi143.bissbc (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:51.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.768+0000 7efd82eef700 1 -- 172.21.15.173:0/3287961330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c0f4540 msgr2=0x7efd7c0f4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:51.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.768+0000 7efd82eef700 1 --2- 172.21.15.173:0/3287961330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c0f4540 0x7efd7c0f4960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7efd78009fb0 tx=0x7efd78009b30).stop 2024-02-19T21:58:51.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.768+0000 7efd82eef700 1 -- 172.21.15.173:0/3287961330 shutdown_connections 2024-02-19T21:58:51.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.768+0000 7efd82eef700 1 --2- 172.21.15.173:0/3287961330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c0f4540 0x7efd7c0f4960 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:51.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.768+0000 7efd82eef700 1 -- 172.21.15.173:0/3287961330 >> 172.21.15.173:0/3287961330 conn(0x7efd7c0effc0 msgr2=0x7efd7c0f23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:51.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.768+0000 7efd82eef700 1 -- 172.21.15.173:0/3287961330 shutdown_connections 2024-02-19T21:58:51.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.768+0000 7efd82eef700 1 -- 172.21.15.173:0/3287961330 wait complete. 2024-02-19T21:58:51.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.769+0000 7efd82eef700 1 Processor -- start 2024-02-19T21:58:51.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.769+0000 7efd82eef700 1 -- start start 2024-02-19T21:58:51.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.769+0000 7efd82eef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c0f4540 0x7efd7c0f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:51.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.769+0000 7efd82eef700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0f54d0 con 0x7efd7c0f4540 2024-02-19T21:58:51.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.769+0000 7efd81eed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c0f4540 0x7efd7c0f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:51.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.769+0000 7efd81eed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c0f4540 0x7efd7c0f7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:57594/0 (socket says 172.21.15.173:57594) 2024-02-19T21:58:51.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.769+0000 7efd81eed700 1 -- 172.21.15.173:0/1782889711 learned_addr learned my addr 172.21.15.173:0/1782889711 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:51.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.769+0000 7efd81eed700 1 -- 172.21.15.173:0/1782889711 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd78005040 con 0x7efd7c0f4540 2024-02-19T21:58:51.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.770+0000 7efd81eed700 1 --2- 172.21.15.173:0/1782889711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c0f4540 0x7efd7c0f7050 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7efd780089f0 tx=0x7efd7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:51.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.770+0000 7efd727fc700 1 -- 172.21.15.173:0/1782889711 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd78009320 con 0x7efd7c0f4540 2024-02-19T21:58:51.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.770+0000 7efd727fc700 1 -- 172.21.15.173:0/1782889711 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd78013380 con 0x7efd7c0f4540 2024-02-19T21:58:51.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.770+0000 7efd82eef700 1 -- 172.21.15.173:0/1782889711 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c0f5670 con 0x7efd7c0f4540 2024-02-19T21:58:51.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.770+0000 7efd727fc700 1 -- 172.21.15.173:0/1782889711 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd7801ed80 con 0x7efd7c0f4540 2024-02-19T21:58:51.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.770+0000 7efd82eef700 1 -- 172.21.15.173:0/1782889711 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c0f5b00 con 0x7efd7c0f4540 2024-02-19T21:58:51.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.771+0000 7efd727fc700 1 -- 172.21.15.173:0/1782889711 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd7801e420 con 0x7efd7c0f4540 2024-02-19T21:58:51.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.771+0000 7efd82eef700 1 -- 172.21.15.173:0/1782889711 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c059100 con 0x7efd7c0f4540 2024-02-19T21:58:51.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.771+0000 7efd727fc700 1 --2- 172.21.15.173:0/1782889711 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd6802ff60 0x7efd68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:51.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.771+0000 7efd727fc700 1 -- 172.21.15.173:0/1782889711 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd78041090 con 0x7efd7c0f4540 2024-02-19T21:58:51.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.771+0000 7efd816ec700 1 --2- 172.21.15.173:0/1782889711 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd6802ff60 0x7efd68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:51.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.771+0000 7efd816ec700 1 --2- 172.21.15.173:0/1782889711 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd6802ff60 0x7efd68032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7efd6c0096a0 tx=0x7efd6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:51.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.776+0000 7efd727fc700 1 -- 172.21.15.173:0/1782889711 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd780095f0 con 0x7efd7c0f4540 2024-02-19T21:58:51.984 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.985+0000 7efd82eef700 1 -- 172.21.15.173:0/1782889711 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd7c0fbe70 con 0x7efd7c0f4540 2024-02-19T21:58:51.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.985+0000 7efd727fc700 1 -- 172.21.15.173:0/1782889711 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efd78004490 con 0x7efd7c0f4540 2024-02-19T21:58:51.985 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:51.985 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:51.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.987+0000 7efd82eef700 1 -- 172.21.15.173:0/1782889711 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd6802ff60 msgr2=0x7efd68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:51.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.987+0000 7efd82eef700 1 --2- 172.21.15.173:0/1782889711 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd6802ff60 0x7efd68032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7efd6c0096a0 tx=0x7efd6c006b40).stop 2024-02-19T21:58:51.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.987+0000 7efd82eef700 1 -- 172.21.15.173:0/1782889711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c0f4540 msgr2=0x7efd7c0f7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:51.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.987+0000 7efd82eef700 1 --2- 172.21.15.173:0/1782889711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c0f4540 0x7efd7c0f7050 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7efd780089f0 tx=0x7efd7800b040).stop 2024-02-19T21:58:51.987 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.987+0000 7efd82eef700 1 -- 172.21.15.173:0/1782889711 shutdown_connections 2024-02-19T21:58:51.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.987+0000 7efd82eef700 1 --2- 172.21.15.173:0/1782889711 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd6802ff60 0x7efd68032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:51.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.987+0000 7efd82eef700 1 --2- 172.21.15.173:0/1782889711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c0f4540 0x7efd7c0f7050 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:51.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.987+0000 7efd82eef700 1 -- 172.21.15.173:0/1782889711 >> 172.21.15.173:0/1782889711 conn(0x7efd7c0effc0 msgr2=0x7efd7c0f0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:51.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.987+0000 7efd82eef700 1 -- 172.21.15.173:0/1782889711 shutdown_connections 2024-02-19T21:58:51.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:51.987+0000 7efd82eef700 1 -- 172.21.15.173:0/1782889711 wait complete. 2024-02-19T21:58:51.988 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:52.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:50.479385+0000 mgr.smithi143.bissbc (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:52.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:51.986693+0000 mon.smithi143 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.173:0/1782889711' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:53.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:53.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:52.479667+0000 mgr.smithi143.bissbc (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:55.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.488+0000 7f1b734d4700 1 -- 172.21.15.173:0/1729758383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b6c0f6510 msgr2=0x7f1b6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.488+0000 7f1b734d4700 1 --2- 172.21.15.173:0/1729758383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b6c0f6510 0x7f1b6c0f6930 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f1b5c009fb0 tx=0x7f1b5c009b30).stop 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.488+0000 7f1b734d4700 1 -- 172.21.15.173:0/1729758383 shutdown_connections 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.488+0000 7f1b734d4700 1 --2- 172.21.15.173:0/1729758383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b6c0f6510 0x7f1b6c0f6930 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.488+0000 7f1b734d4700 1 -- 172.21.15.173:0/1729758383 >> 172.21.15.173:0/1729758383 conn(0x7f1b6c0f1c70 msgr2=0x7f1b6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.488+0000 7f1b734d4700 1 -- 172.21.15.173:0/1729758383 shutdown_connections 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.488+0000 7f1b734d4700 1 -- 172.21.15.173:0/1729758383 wait complete. 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.489+0000 7f1b734d4700 1 Processor -- start 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.489+0000 7f1b734d4700 1 -- start start 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.489+0000 7f1b734d4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b6c0f6510 0x7f1b6c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.489+0000 7f1b734d4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c106cd0 con 0x7f1b6c0f6510 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.489+0000 7f1b71270700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b6c0f6510 0x7f1b6c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.489+0000 7f1b71270700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b6c0f6510 0x7f1b6c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:57610/0 (socket says 172.21.15.173:57610) 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.489+0000 7f1b71270700 1 -- 172.21.15.173:0/2224613662 learned_addr learned my addr 172.21.15.173:0/2224613662 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.489+0000 7f1b71270700 1 -- 172.21.15.173:0/2224613662 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c005040 con 0x7f1b6c0f6510 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.489+0000 7f1b71270700 1 --2- 172.21.15.173:0/2224613662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b6c0f6510 0x7f1b6c1067e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f1b5c0089f0 tx=0x7f1b5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.490+0000 7f1b61ffb700 1 -- 172.21.15.173:0/2224613662 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b5c009320 con 0x7f1b6c0f6510 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.490+0000 7f1b734d4700 1 -- 172.21.15.173:0/2224613662 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c106ed0 con 0x7f1b6c0f6510 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.490+0000 7f1b61ffb700 1 -- 172.21.15.173:0/2224613662 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b5c013380 con 0x7f1b6c0f6510 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.490+0000 7f1b61ffb700 1 -- 172.21.15.173:0/2224613662 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b5c01ed80 con 0x7f1b6c0f6510 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.490+0000 7f1b734d4700 1 -- 172.21.15.173:0/2224613662 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c107360 con 0x7f1b6c0f6510 2024-02-19T21:58:55.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.491+0000 7f1b61ffb700 1 -- 172.21.15.173:0/2224613662 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b5c014420 con 0x7f1b6c0f6510 2024-02-19T21:58:55.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.491+0000 7f1b61ffb700 1 --2- 172.21.15.173:0/2224613662 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1b5802ff60 0x7f1b58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:55.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.491+0000 7f1b734d4700 1 -- 172.21.15.173:0/2224613662 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6c045ad0 con 0x7f1b6c0f6510 2024-02-19T21:58:55.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.491+0000 7f1b61ffb700 1 -- 172.21.15.173:0/2224613662 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b5c040e70 con 0x7f1b6c0f6510 2024-02-19T21:58:55.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.491+0000 7f1b70a6f700 1 --2- 172.21.15.173:0/2224613662 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1b5802ff60 0x7f1b58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:55.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.491+0000 7f1b70a6f700 1 --2- 172.21.15.173:0/2224613662 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1b5802ff60 0x7f1b58032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f1b680096a0 tx=0x7f1b68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:55.495 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.496+0000 7f1b61ffb700 1 -- 172.21.15.173:0/2224613662 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b5c01c910 con 0x7f1b6c0f6510 2024-02-19T21:58:55.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.701+0000 7f1b734d4700 1 -- 172.21.15.173:0/2224613662 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b6c1081b0 con 0x7f1b6c0f6510 2024-02-19T21:58:55.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.701+0000 7f1b61ffb700 1 -- 172.21.15.173:0/2224613662 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1b5c01c320 con 0x7f1b6c0f6510 2024-02-19T21:58:55.701 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:55.701 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:55.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.703+0000 7f1b734d4700 1 -- 172.21.15.173:0/2224613662 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1b5802ff60 msgr2=0x7f1b58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:55.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.703+0000 7f1b734d4700 1 --2- 172.21.15.173:0/2224613662 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1b5802ff60 0x7f1b58032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f1b680096a0 tx=0x7f1b68006b40).stop 2024-02-19T21:58:55.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.703+0000 7f1b734d4700 1 -- 172.21.15.173:0/2224613662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b6c0f6510 msgr2=0x7f1b6c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:55.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.703+0000 7f1b734d4700 1 --2- 172.21.15.173:0/2224613662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b6c0f6510 0x7f1b6c1067e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f1b5c0089f0 tx=0x7f1b5c00b040).stop 2024-02-19T21:58:55.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.703+0000 7f1b734d4700 1 -- 172.21.15.173:0/2224613662 shutdown_connections 2024-02-19T21:58:55.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.703+0000 7f1b734d4700 1 --2- 172.21.15.173:0/2224613662 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1b5802ff60 0x7f1b58032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:55.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.703+0000 7f1b734d4700 1 --2- 172.21.15.173:0/2224613662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b6c0f6510 0x7f1b6c1067e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:55.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.703+0000 7f1b734d4700 1 -- 172.21.15.173:0/2224613662 >> 172.21.15.173:0/2224613662 conn(0x7f1b6c0f1c70 msgr2=0x7f1b6c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:55.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.703+0000 7f1b734d4700 1 -- 172.21.15.173:0/2224613662 shutdown_connections 2024-02-19T21:58:55.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:55.703+0000 7f1b734d4700 1 -- 172.21.15.173:0/2224613662 wait complete. 2024-02-19T21:58:55.704 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:58:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:54.479932+0000 mgr.smithi143.bissbc (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:55.702486+0000 mon.smithi143 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.173:0/2224613662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:57.385 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:57.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:58:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:56.480311+0000 mgr.smithi143.bissbc (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:59.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.411+0000 7f96b8eaf700 1 -- 172.21.15.173:0/1921514058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b40f5410 msgr2=0x7f96b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:59.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.411+0000 7f96b8eaf700 1 --2- 172.21.15.173:0/1921514058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b40f5410 0x7f96b40f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f969c009fb0 tx=0x7f969c009b30).stop 2024-02-19T21:58:59.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.412+0000 7f96b8eaf700 1 -- 172.21.15.173:0/1921514058 shutdown_connections 2024-02-19T21:58:59.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.412+0000 7f96b8eaf700 1 --2- 172.21.15.173:0/1921514058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b40f5410 0x7f96b40f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:59.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.412+0000 7f96b8eaf700 1 -- 172.21.15.173:0/1921514058 >> 172.21.15.173:0/1921514058 conn(0x7f96b40f0b70 msgr2=0x7f96b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:59.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.412+0000 7f96b8eaf700 1 -- 172.21.15.173:0/1921514058 shutdown_connections 2024-02-19T21:58:59.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.412+0000 7f96b8eaf700 1 -- 172.21.15.173:0/1921514058 wait complete. 2024-02-19T21:58:59.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.412+0000 7f96b8eaf700 1 Processor -- start 2024-02-19T21:58:59.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.412+0000 7f96b8eaf700 1 -- start start 2024-02-19T21:58:59.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.413+0000 7f96b8eaf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b40f5410 0x7f96b4105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:59.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.413+0000 7f96b8eaf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b4105c00 con 0x7f96b40f5410 2024-02-19T21:58:59.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.413+0000 7f96b259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b40f5410 0x7f96b4105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:59.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.413+0000 7f96b259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b40f5410 0x7f96b4105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:57634/0 (socket says 172.21.15.173:57634) 2024-02-19T21:58:59.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.413+0000 7f96b259c700 1 -- 172.21.15.173:0/2150993333 learned_addr learned my addr 172.21.15.173:0/2150993333 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:58:59.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.413+0000 7f96b259c700 1 -- 172.21.15.173:0/2150993333 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c005040 con 0x7f96b40f5410 2024-02-19T21:58:59.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.413+0000 7f96b259c700 1 --2- 172.21.15.173:0/2150993333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b40f5410 0x7f96b4105710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f969c005010 tx=0x7f969c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:59.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.413+0000 7f96aaffd700 1 -- 172.21.15.173:0/2150993333 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f969c009320 con 0x7f96b40f5410 2024-02-19T21:58:59.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.414+0000 7f96b8eaf700 1 -- 172.21.15.173:0/2150993333 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b4105e00 con 0x7f96b40f5410 2024-02-19T21:58:59.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.414+0000 7f96aaffd700 1 -- 172.21.15.173:0/2150993333 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f969c013380 con 0x7f96b40f5410 2024-02-19T21:58:59.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.414+0000 7f96aaffd700 1 -- 172.21.15.173:0/2150993333 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f969c014e50 con 0x7f96b40f5410 2024-02-19T21:58:59.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.414+0000 7f96b8eaf700 1 -- 172.21.15.173:0/2150993333 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b4106290 con 0x7f96b40f5410 2024-02-19T21:58:59.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.414+0000 7f96aaffd700 1 -- 172.21.15.173:0/2150993333 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f969c0263a0 con 0x7f96b40f5410 2024-02-19T21:58:59.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.414+0000 7f96aaffd700 1 --2- 172.21.15.173:0/2150993333 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96a002ff10 0x7f96a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:59.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.414+0000 7f96aaffd700 1 -- 172.21.15.173:0/2150993333 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f969c040fb0 con 0x7f96b40f5410 2024-02-19T21:58:59.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.415+0000 7f96b8eaf700 1 -- 172.21.15.173:0/2150993333 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b4045ad0 con 0x7f96b40f5410 2024-02-19T21:58:59.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.415+0000 7f96b1d9b700 1 --2- 172.21.15.173:0/2150993333 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96a002ff10 0x7f96a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:59.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.415+0000 7f96b1d9b700 1 --2- 172.21.15.173:0/2150993333 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96a002ff10 0x7f96a00323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f96a40096a0 tx=0x7f96a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:59.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.420+0000 7f96aaffd700 1 -- 172.21.15.173:0/2150993333 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f969c012920 con 0x7f96b40f5410 2024-02-19T21:58:59.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.627+0000 7f96b8eaf700 1 -- 172.21.15.173:0/2150993333 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96b4106fd0 con 0x7f96b40f5410 2024-02-19T21:58:59.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.627+0000 7f96aaffd700 1 -- 172.21.15.173:0/2150993333 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f969c02b020 con 0x7f96b40f5410 2024-02-19T21:58:59.627 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:58:59.627 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.629+0000 7f96b8eaf700 1 -- 172.21.15.173:0/2150993333 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96a002ff10 msgr2=0x7f96a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.629+0000 7f96b8eaf700 1 --2- 172.21.15.173:0/2150993333 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96a002ff10 0x7f96a00323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f96a40096a0 tx=0x7f96a4006b40).stop 2024-02-19T21:58:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.629+0000 7f96b8eaf700 1 -- 172.21.15.173:0/2150993333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b40f5410 msgr2=0x7f96b4105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.629+0000 7f96b8eaf700 1 --2- 172.21.15.173:0/2150993333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b40f5410 0x7f96b4105710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f969c005010 tx=0x7f969c00b040).stop 2024-02-19T21:58:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.629+0000 7f96b8eaf700 1 -- 172.21.15.173:0/2150993333 shutdown_connections 2024-02-19T21:58:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.629+0000 7f96b8eaf700 1 --2- 172.21.15.173:0/2150993333 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96a002ff10 0x7f96a00323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.629+0000 7f96b8eaf700 1 --2- 172.21.15.173:0/2150993333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b40f5410 0x7f96b4105710 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.629+0000 7f96b8eaf700 1 -- 172.21.15.173:0/2150993333 >> 172.21.15.173:0/2150993333 conn(0x7f96b40f0b70 msgr2=0x7f96b40f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:59.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.629+0000 7f96b8eaf700 1 -- 172.21.15.173:0/2150993333 shutdown_connections 2024-02-19T21:58:59.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:58:59.629+0000 7f96b8eaf700 1 -- 172.21.15.173:0/2150993333 wait complete. 2024-02-19T21:58:59.630 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:58:58.480668+0000 mgr.smithi143.bissbc (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:58:59.628579+0000 mon.smithi143 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.173:0/2150993333' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:01.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:01.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:00.480945+0000 mgr.smithi143.bissbc (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:03.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.379+0000 7f7d71544700 1 -- 172.21.15.173:0/2693301344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6c0f51a0 msgr2=0x7f7d6c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:03.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.379+0000 7f7d71544700 1 --2- 172.21.15.173:0/2693301344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6c0f51a0 0x7f7d6c0f55c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f7d5c009fb0 tx=0x7f7d5c009b30).stop 2024-02-19T21:59:03.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.379+0000 7f7d71544700 1 -- 172.21.15.173:0/2693301344 shutdown_connections 2024-02-19T21:59:03.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.379+0000 7f7d71544700 1 --2- 172.21.15.173:0/2693301344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6c0f51a0 0x7f7d6c0f55c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:03.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.379+0000 7f7d71544700 1 -- 172.21.15.173:0/2693301344 >> 172.21.15.173:0/2693301344 conn(0x7f7d6c0f09a0 msgr2=0x7f7d6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:03.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.379+0000 7f7d71544700 1 -- 172.21.15.173:0/2693301344 shutdown_connections 2024-02-19T21:59:03.379 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.379+0000 7f7d71544700 1 -- 172.21.15.173:0/2693301344 wait complete. 2024-02-19T21:59:03.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.379+0000 7f7d71544700 1 Processor -- start 2024-02-19T21:59:03.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.379+0000 7f7d71544700 1 -- start start 2024-02-19T21:59:03.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.380+0000 7f7d71544700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6c0f51a0 0x7f7d6c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:03.380 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.380+0000 7f7d71544700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c106ac0 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.381 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.380+0000 7f7d6bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6c0f51a0 0x7f7d6c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.380+0000 7f7d6bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6c0f51a0 0x7f7d6c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43132/0 (socket says 172.21.15.173:43132) 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.380+0000 7f7d6bfff700 1 -- 172.21.15.173:0/1051789129 learned_addr learned my addr 172.21.15.173:0/1051789129 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.380+0000 7f7d6bfff700 1 -- 172.21.15.173:0/1051789129 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c005040 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.380+0000 7f7d6bfff700 1 --2- 172.21.15.173:0/1051789129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6c0f51a0 0x7f7d6c1065d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f7d5c009830 tx=0x7f7d5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.380+0000 7f7d68ff9700 1 -- 172.21.15.173:0/1051789129 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d5c009320 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.380+0000 7f7d71544700 1 -- 172.21.15.173:0/1051789129 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6c106cc0 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.380+0000 7f7d68ff9700 1 -- 172.21.15.173:0/1051789129 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d5c013380 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.380+0000 7f7d68ff9700 1 -- 172.21.15.173:0/1051789129 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d5c014420 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.381+0000 7f7d71544700 1 -- 172.21.15.173:0/1051789129 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6c107150 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.381+0000 7f7d68ff9700 1 -- 172.21.15.173:0/1051789129 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d5c014580 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.381+0000 7f7d68ff9700 1 --2- 172.21.15.173:0/1051789129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7d5402ff60 0x7f7d54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.382+0000 7f7d68ff9700 1 -- 172.21.15.173:0/1051789129 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d5c040eb0 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.382+0000 7f7d71544700 1 -- 172.21.15.173:0/1051789129 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6c045ad0 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.382+0000 7f7d6b7fe700 1 --2- 172.21.15.173:0/1051789129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7d5402ff60 0x7f7d54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:03.382 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.382+0000 7f7d6b7fe700 1 --2- 172.21.15.173:0/1051789129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7d5402ff60 0x7f7d54032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f7d600096a0 tx=0x7f7d60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:03.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.387+0000 7f7d68ff9700 1 -- 172.21.15.173:0/1051789129 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d5c0273f0 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.594+0000 7f7d71544700 1 -- 172.21.15.173:0/1051789129 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d6c108660 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.595+0000 7f7d68ff9700 1 -- 172.21.15.173:0/1051789129 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7d5c01c890 con 0x7f7d6c0f51a0 2024-02-19T21:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:03.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.597+0000 7f7d71544700 1 -- 172.21.15.173:0/1051789129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7d5402ff60 msgr2=0x7f7d54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:03.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.597+0000 7f7d71544700 1 --2- 172.21.15.173:0/1051789129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7d5402ff60 0x7f7d54032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f7d600096a0 tx=0x7f7d60006b40).stop 2024-02-19T21:59:03.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.597+0000 7f7d71544700 1 -- 172.21.15.173:0/1051789129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6c0f51a0 msgr2=0x7f7d6c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:03.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.597+0000 7f7d71544700 1 --2- 172.21.15.173:0/1051789129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6c0f51a0 0x7f7d6c1065d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f7d5c009830 tx=0x7f7d5c00b040).stop 2024-02-19T21:59:03.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.597+0000 7f7d71544700 1 -- 172.21.15.173:0/1051789129 shutdown_connections 2024-02-19T21:59:03.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.597+0000 7f7d71544700 1 --2- 172.21.15.173:0/1051789129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7d5402ff60 0x7f7d54032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:03.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.597+0000 7f7d71544700 1 --2- 172.21.15.173:0/1051789129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6c0f51a0 0x7f7d6c1065d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:03.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.597+0000 7f7d71544700 1 -- 172.21.15.173:0/1051789129 >> 172.21.15.173:0/1051789129 conn(0x7f7d6c0f09a0 msgr2=0x7f7d6c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:03.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.597+0000 7f7d71544700 1 -- 172.21.15.173:0/1051789129 shutdown_connections 2024-02-19T21:59:03.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:03.597+0000 7f7d71544700 1 -- 172.21.15.173:0/1051789129 wait complete. 2024-02-19T21:59:03.598 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:02.481247+0000 mgr.smithi143.bissbc (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:03.595800+0000 mon.smithi143 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.173:0/1051789129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:05.332 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:05.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:04.481660+0000 mgr.smithi143.bissbc (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:07.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.304+0000 7ff439bd2700 1 -- 172.21.15.173:0/408014790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4340f6510 msgr2=0x7ff4340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.304+0000 7ff439bd2700 1 --2- 172.21.15.173:0/408014790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4340f6510 0x7ff4340f6930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7ff428009fb0 tx=0x7ff428009b30).stop 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.304+0000 7ff439bd2700 1 -- 172.21.15.173:0/408014790 shutdown_connections 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.304+0000 7ff439bd2700 1 --2- 172.21.15.173:0/408014790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4340f6510 0x7ff4340f6930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.304+0000 7ff439bd2700 1 -- 172.21.15.173:0/408014790 >> 172.21.15.173:0/408014790 conn(0x7ff4340f1c70 msgr2=0x7ff4340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.304+0000 7ff439bd2700 1 -- 172.21.15.173:0/408014790 shutdown_connections 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.304+0000 7ff439bd2700 1 -- 172.21.15.173:0/408014790 wait complete. 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.304+0000 7ff439bd2700 1 Processor -- start 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.305+0000 7ff439bd2700 1 -- start start 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.305+0000 7ff439bd2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4340f6510 0x7ff4341067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.305+0000 7ff439bd2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff434106cd0 con 0x7ff4340f6510 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.305+0000 7ff4337fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4340f6510 0x7ff4341067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:07.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.305+0000 7ff4337fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4340f6510 0x7ff4341067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43152/0 (socket says 172.21.15.173:43152) 2024-02-19T21:59:07.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.305+0000 7ff4337fe700 1 -- 172.21.15.173:0/2319845434 learned_addr learned my addr 172.21.15.173:0/2319845434 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:07.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.305+0000 7ff4337fe700 1 -- 172.21.15.173:0/2319845434 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff428005040 con 0x7ff4340f6510 2024-02-19T21:59:07.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.306+0000 7ff4337fe700 1 --2- 172.21.15.173:0/2319845434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4340f6510 0x7ff4341067e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7ff428005010 tx=0x7ff42800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:07.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.306+0000 7ff41bfff700 1 -- 172.21.15.173:0/2319845434 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff428009320 con 0x7ff4340f6510 2024-02-19T21:59:07.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.306+0000 7ff439bd2700 1 -- 172.21.15.173:0/2319845434 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff434106ed0 con 0x7ff4340f6510 2024-02-19T21:59:07.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.306+0000 7ff41bfff700 1 -- 172.21.15.173:0/2319845434 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff428013380 con 0x7ff4340f6510 2024-02-19T21:59:07.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.306+0000 7ff41bfff700 1 -- 172.21.15.173:0/2319845434 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff428014e50 con 0x7ff4340f6510 2024-02-19T21:59:07.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.306+0000 7ff439bd2700 1 -- 172.21.15.173:0/2319845434 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff434107360 con 0x7ff4340f6510 2024-02-19T21:59:07.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.307+0000 7ff41bfff700 1 -- 172.21.15.173:0/2319845434 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4280263a0 con 0x7ff4340f6510 2024-02-19T21:59:07.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.307+0000 7ff41bfff700 1 --2- 172.21.15.173:0/2319845434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff41c02ff10 0x7ff41c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:07.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.307+0000 7ff41bfff700 1 -- 172.21.15.173:0/2319845434 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff428040f30 con 0x7ff4340f6510 2024-02-19T21:59:07.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.307+0000 7ff439bd2700 1 -- 172.21.15.173:0/2319845434 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff434045ad0 con 0x7ff4340f6510 2024-02-19T21:59:07.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.307+0000 7ff432ffd700 1 --2- 172.21.15.173:0/2319845434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff41c02ff10 0x7ff41c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:07.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.308+0000 7ff432ffd700 1 --2- 172.21.15.173:0/2319845434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff41c02ff10 0x7ff41c0323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7ff4240096a0 tx=0x7ff424006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:07.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.312+0000 7ff41bfff700 1 -- 172.21.15.173:0/2319845434 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff428012920 con 0x7ff4340f6510 2024-02-19T21:59:07.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.518+0000 7ff439bd2700 1 -- 172.21.15.173:0/2319845434 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4341080b0 con 0x7ff4340f6510 2024-02-19T21:59:07.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.519+0000 7ff41bfff700 1 -- 172.21.15.173:0/2319845434 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff42802b020 con 0x7ff4340f6510 2024-02-19T21:59:07.519 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:07.519 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:07.520 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.521+0000 7ff439bd2700 1 -- 172.21.15.173:0/2319845434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff41c02ff10 msgr2=0x7ff41c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:07.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.521+0000 7ff439bd2700 1 --2- 172.21.15.173:0/2319845434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff41c02ff10 0x7ff41c0323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7ff4240096a0 tx=0x7ff424006b40).stop 2024-02-19T21:59:07.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.521+0000 7ff439bd2700 1 -- 172.21.15.173:0/2319845434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4340f6510 msgr2=0x7ff4341067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:07.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.521+0000 7ff439bd2700 1 --2- 172.21.15.173:0/2319845434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4340f6510 0x7ff4341067e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7ff428005010 tx=0x7ff42800b040).stop 2024-02-19T21:59:07.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.521+0000 7ff439bd2700 1 -- 172.21.15.173:0/2319845434 shutdown_connections 2024-02-19T21:59:07.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.521+0000 7ff439bd2700 1 --2- 172.21.15.173:0/2319845434 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff41c02ff10 0x7ff41c0323d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:07.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.521+0000 7ff439bd2700 1 --2- 172.21.15.173:0/2319845434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4340f6510 0x7ff4341067e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:07.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.521+0000 7ff439bd2700 1 -- 172.21.15.173:0/2319845434 >> 172.21.15.173:0/2319845434 conn(0x7ff4340f1c70 msgr2=0x7ff4340f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:07.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.521+0000 7ff439bd2700 1 -- 172.21.15.173:0/2319845434 shutdown_connections 2024-02-19T21:59:07.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:07.521+0000 7ff439bd2700 1 -- 172.21.15.173:0/2319845434 wait complete. 2024-02-19T21:59:07.522 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:06.481918+0000 mgr.smithi143.bissbc (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:07.520233+0000 mon.smithi143 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.173:0/2319845434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:09.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:09.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:08.482159+0000 mgr.smithi143.bissbc (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:11.333 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.329+0000 7fdb0ef51700 1 -- 172.21.15.173:0/3843810900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb080f6510 msgr2=0x7fdb080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:11.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.329+0000 7fdb0ef51700 1 --2- 172.21.15.173:0/3843810900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb080f6510 0x7fdb080f6930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fdafc009fb0 tx=0x7fdafc009b30).stop 2024-02-19T21:59:11.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.329+0000 7fdb0ef51700 1 -- 172.21.15.173:0/3843810900 shutdown_connections 2024-02-19T21:59:11.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.329+0000 7fdb0ef51700 1 --2- 172.21.15.173:0/3843810900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb080f6510 0x7fdb080f6930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:11.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.329+0000 7fdb0ef51700 1 -- 172.21.15.173:0/3843810900 >> 172.21.15.173:0/3843810900 conn(0x7fdb080f1c70 msgr2=0x7fdb080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:11.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.329+0000 7fdb0ef51700 1 -- 172.21.15.173:0/3843810900 shutdown_connections 2024-02-19T21:59:11.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.329+0000 7fdb0ef51700 1 -- 172.21.15.173:0/3843810900 wait complete. 2024-02-19T21:59:11.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.330+0000 7fdb0ef51700 1 Processor -- start 2024-02-19T21:59:11.334 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.330+0000 7fdb0ef51700 1 -- start start 2024-02-19T21:59:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.330+0000 7fdb0ef51700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb080f6510 0x7fdb081067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.330+0000 7fdb0ef51700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb08106cd0 con 0x7fdb080f6510 2024-02-19T21:59:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.330+0000 7fdb0cced700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb080f6510 0x7fdb081067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.330+0000 7fdb0cced700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb080f6510 0x7fdb081067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:35256/0 (socket says 172.21.15.173:35256) 2024-02-19T21:59:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.330+0000 7fdb0cced700 1 -- 172.21.15.173:0/2710217552 learned_addr learned my addr 172.21.15.173:0/2710217552 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.331+0000 7fdb0cced700 1 -- 172.21.15.173:0/2710217552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc005040 con 0x7fdb080f6510 2024-02-19T21:59:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.331+0000 7fdb0cced700 1 --2- 172.21.15.173:0/2710217552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb080f6510 0x7fdb081067e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fdafc0089f0 tx=0x7fdafc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.331+0000 7fdb057fa700 1 -- 172.21.15.173:0/2710217552 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdafc009320 con 0x7fdb080f6510 2024-02-19T21:59:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.331+0000 7fdb0ef51700 1 -- 172.21.15.173:0/2710217552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb08106ed0 con 0x7fdb080f6510 2024-02-19T21:59:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.331+0000 7fdb057fa700 1 -- 172.21.15.173:0/2710217552 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdafc013380 con 0x7fdb080f6510 2024-02-19T21:59:11.335 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.331+0000 7fdb057fa700 1 -- 172.21.15.173:0/2710217552 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdafc01ed80 con 0x7fdb080f6510 2024-02-19T21:59:11.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.331+0000 7fdb0ef51700 1 -- 172.21.15.173:0/2710217552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb08107360 con 0x7fdb080f6510 2024-02-19T21:59:11.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.332+0000 7fdb057fa700 1 -- 172.21.15.173:0/2710217552 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdafc01e420 con 0x7fdb080f6510 2024-02-19T21:59:11.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.332+0000 7fdb0ef51700 1 -- 172.21.15.173:0/2710217552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb08045ad0 con 0x7fdb080f6510 2024-02-19T21:59:11.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.332+0000 7fdb057fa700 1 --2- 172.21.15.173:0/2710217552 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdaf802ffb0 0x7fdaf8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:11.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.332+0000 7fdb057fa700 1 -- 172.21.15.173:0/2710217552 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdafc040f30 con 0x7fdb080f6510 2024-02-19T21:59:11.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.332+0000 7fdb07fff700 1 --2- 172.21.15.173:0/2710217552 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdaf802ffb0 0x7fdaf8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:11.336 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.333+0000 7fdb07fff700 1 --2- 172.21.15.173:0/2710217552 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdaf802ffb0 0x7fdaf8032470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdaf40096a0 tx=0x7fdaf4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:11.337 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.337+0000 7fdb057fa700 1 -- 172.21.15.173:0/2710217552 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdafc0095f0 con 0x7fdb080f6510 2024-02-19T21:59:11.544 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.544+0000 7fdb0ef51700 1 -- 172.21.15.173:0/2710217552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb08108210 con 0x7fdb080f6510 2024-02-19T21:59:11.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.544+0000 7fdb057fa700 1 -- 172.21.15.173:0/2710217552 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdafc01cb10 con 0x7fdb080f6510 2024-02-19T21:59:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:11.545 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:11.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.546+0000 7fdb0ef51700 1 -- 172.21.15.173:0/2710217552 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdaf802ffb0 msgr2=0x7fdaf8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:11.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.546+0000 7fdb0ef51700 1 --2- 172.21.15.173:0/2710217552 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdaf802ffb0 0x7fdaf8032470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdaf40096a0 tx=0x7fdaf4006b40).stop 2024-02-19T21:59:11.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.546+0000 7fdb0ef51700 1 -- 172.21.15.173:0/2710217552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb080f6510 msgr2=0x7fdb081067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:11.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.546+0000 7fdb0ef51700 1 --2- 172.21.15.173:0/2710217552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb080f6510 0x7fdb081067e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fdafc0089f0 tx=0x7fdafc00b040).stop 2024-02-19T21:59:11.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.546+0000 7fdb0ef51700 1 -- 172.21.15.173:0/2710217552 shutdown_connections 2024-02-19T21:59:11.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.546+0000 7fdb0ef51700 1 --2- 172.21.15.173:0/2710217552 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdaf802ffb0 0x7fdaf8032470 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:11.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.546+0000 7fdb0ef51700 1 --2- 172.21.15.173:0/2710217552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb080f6510 0x7fdb081067e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:11.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.546+0000 7fdb0ef51700 1 -- 172.21.15.173:0/2710217552 >> 172.21.15.173:0/2710217552 conn(0x7fdb080f1c70 msgr2=0x7fdb080f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:11.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.546+0000 7fdb0ef51700 1 -- 172.21.15.173:0/2710217552 shutdown_connections 2024-02-19T21:59:11.546 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:11.546+0000 7fdb0ef51700 1 -- 172.21.15.173:0/2710217552 wait complete. 2024-02-19T21:59:11.547 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:12.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:10.482493+0000 mgr.smithi143.bissbc (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:12.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:11.545561+0000 mon.smithi143 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.173:0/2710217552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:13.267 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:13.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:12.482728+0000 mgr.smithi143.bissbc (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:15.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.289+0000 7fbb11a84700 1 -- 172.21.15.173:0/2791580760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c060280 msgr2=0x7fbb0c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:15.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.289+0000 7fbb11a84700 1 --2- 172.21.15.173:0/2791580760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c060280 0x7fbb0c0f90f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fbb00009fb0 tx=0x7fbb00009b30).stop 2024-02-19T21:59:15.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.289+0000 7fbb11a84700 1 -- 172.21.15.173:0/2791580760 shutdown_connections 2024-02-19T21:59:15.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.289+0000 7fbb11a84700 1 --2- 172.21.15.173:0/2791580760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c060280 0x7fbb0c0f90f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:15.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.289+0000 7fbb11a84700 1 -- 172.21.15.173:0/2791580760 >> 172.21.15.173:0/2791580760 conn(0x7fbb0c0f0b70 msgr2=0x7fbb0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:15.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.289+0000 7fbb11a84700 1 -- 172.21.15.173:0/2791580760 shutdown_connections 2024-02-19T21:59:15.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.289+0000 7fbb11a84700 1 -- 172.21.15.173:0/2791580760 wait complete. 2024-02-19T21:59:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.290+0000 7fbb11a84700 1 Processor -- start 2024-02-19T21:59:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.290+0000 7fbb11a84700 1 -- start start 2024-02-19T21:59:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.290+0000 7fbb11a84700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c060280 0x7fbb0c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.290+0000 7fbb11a84700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c106c70 con 0x7fbb0c060280 2024-02-19T21:59:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.290+0000 7fbb0b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c060280 0x7fbb0c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.290+0000 7fbb0b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c060280 0x7fbb0c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:35282/0 (socket says 172.21.15.173:35282) 2024-02-19T21:59:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.290+0000 7fbb0b7fe700 1 -- 172.21.15.173:0/556575999 learned_addr learned my addr 172.21.15.173:0/556575999 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.291+0000 7fbb0b7fe700 1 -- 172.21.15.173:0/556575999 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb00005040 con 0x7fbb0c060280 2024-02-19T21:59:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.291+0000 7fbb0b7fe700 1 --2- 172.21.15.173:0/556575999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c060280 0x7fbb0c106780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fbb000089f0 tx=0x7fbb0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.291+0000 7fbaf3fff700 1 -- 172.21.15.173:0/556575999 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb00009320 con 0x7fbb0c060280 2024-02-19T21:59:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.291+0000 7fbb11a84700 1 -- 172.21.15.173:0/556575999 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0c106e70 con 0x7fbb0c060280 2024-02-19T21:59:15.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.291+0000 7fbaf3fff700 1 -- 172.21.15.173:0/556575999 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb00013380 con 0x7fbb0c060280 2024-02-19T21:59:15.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.291+0000 7fbaf3fff700 1 -- 172.21.15.173:0/556575999 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb00014e20 con 0x7fbb0c060280 2024-02-19T21:59:15.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.291+0000 7fbb11a84700 1 -- 172.21.15.173:0/556575999 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0c107300 con 0x7fbb0c060280 2024-02-19T21:59:15.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.292+0000 7fbaf3fff700 1 -- 172.21.15.173:0/556575999 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb00014420 con 0x7fbb0c060280 2024-02-19T21:59:15.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.292+0000 7fbb11a84700 1 -- 172.21.15.173:0/556575999 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0c045ad0 con 0x7fbb0c060280 2024-02-19T21:59:15.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.292+0000 7fbaf3fff700 1 --2- 172.21.15.173:0/556575999 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbaf402ff60 0x7fbaf4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:15.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.292+0000 7fbaf3fff700 1 -- 172.21.15.173:0/556575999 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb00040fc0 con 0x7fbb0c060280 2024-02-19T21:59:15.293 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.292+0000 7fbb0affd700 1 --2- 172.21.15.173:0/556575999 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbaf402ff60 0x7fbaf4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:15.294 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.293+0000 7fbb0affd700 1 --2- 172.21.15.173:0/556575999 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbaf402ff60 0x7fbaf4032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fbafc0096a0 tx=0x7fbafc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:15.297 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.297+0000 7fbaf3fff700 1 -- 172.21.15.173:0/556575999 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb000095f0 con 0x7fbb0c060280 2024-02-19T21:59:15.503 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.503+0000 7fbb11a84700 1 -- 172.21.15.173:0/556575999 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb0c1081b0 con 0x7fbb0c060280 2024-02-19T21:59:15.504 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.504+0000 7fbaf3fff700 1 -- 172.21.15.173:0/556575999 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbb0001db70 con 0x7fbb0c060280 2024-02-19T21:59:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:15.504 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:15.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.506+0000 7fbb11a84700 1 -- 172.21.15.173:0/556575999 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbaf402ff60 msgr2=0x7fbaf4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:15.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.506+0000 7fbb11a84700 1 --2- 172.21.15.173:0/556575999 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbaf402ff60 0x7fbaf4032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fbafc0096a0 tx=0x7fbafc006b40).stop 2024-02-19T21:59:15.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.506+0000 7fbb11a84700 1 -- 172.21.15.173:0/556575999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c060280 msgr2=0x7fbb0c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:15.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.506+0000 7fbb11a84700 1 --2- 172.21.15.173:0/556575999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c060280 0x7fbb0c106780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fbb000089f0 tx=0x7fbb0000b040).stop 2024-02-19T21:59:15.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.506+0000 7fbb11a84700 1 -- 172.21.15.173:0/556575999 shutdown_connections 2024-02-19T21:59:15.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.506+0000 7fbb11a84700 1 --2- 172.21.15.173:0/556575999 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbaf402ff60 0x7fbaf4032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:15.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.506+0000 7fbb11a84700 1 --2- 172.21.15.173:0/556575999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c060280 0x7fbb0c106780 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:15.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.506+0000 7fbb11a84700 1 -- 172.21.15.173:0/556575999 >> 172.21.15.173:0/556575999 conn(0x7fbb0c0f0b70 msgr2=0x7fbb0c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:15.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.506+0000 7fbb11a84700 1 -- 172.21.15.173:0/556575999 shutdown_connections 2024-02-19T21:59:15.506 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:15.506+0000 7fbb11a84700 1 -- 172.21.15.173:0/556575999 wait complete. 2024-02-19T21:59:15.507 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:14.483029+0000 mgr.smithi143.bissbc (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:15.505326+0000 mon.smithi143 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.173:0/556575999' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:17.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:17.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:16.483301+0000 mgr.smithi143.bissbc (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:19.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.203+0000 7fefe4e2e700 1 -- 172.21.15.173:0/3765882762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe00f6510 msgr2=0x7fefe00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:19.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.203+0000 7fefe4e2e700 1 --2- 172.21.15.173:0/3765882762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe00f6510 0x7fefe00f6930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fefd4009fb0 tx=0x7fefd4009b30).stop 2024-02-19T21:59:19.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.203+0000 7fefe4e2e700 1 -- 172.21.15.173:0/3765882762 shutdown_connections 2024-02-19T21:59:19.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.203+0000 7fefe4e2e700 1 --2- 172.21.15.173:0/3765882762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe00f6510 0x7fefe00f6930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:19.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.203+0000 7fefe4e2e700 1 -- 172.21.15.173:0/3765882762 >> 172.21.15.173:0/3765882762 conn(0x7fefe00f1c70 msgr2=0x7fefe00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:19.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.203+0000 7fefe4e2e700 1 -- 172.21.15.173:0/3765882762 shutdown_connections 2024-02-19T21:59:19.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.203+0000 7fefe4e2e700 1 -- 172.21.15.173:0/3765882762 wait complete. 2024-02-19T21:59:19.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.204+0000 7fefe4e2e700 1 Processor -- start 2024-02-19T21:59:19.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.204+0000 7fefe4e2e700 1 -- start start 2024-02-19T21:59:19.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.204+0000 7fefe4e2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe00f6510 0x7fefe01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:19.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.204+0000 7fefe4e2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe0106cd0 con 0x7fefe00f6510 2024-02-19T21:59:19.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.204+0000 7fefde59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe00f6510 0x7fefe01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.204+0000 7fefde59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe00f6510 0x7fefe01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:35310/0 (socket says 172.21.15.173:35310) 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.204+0000 7fefde59c700 1 -- 172.21.15.173:0/469270250 learned_addr learned my addr 172.21.15.173:0/469270250 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.205+0000 7fefde59c700 1 -- 172.21.15.173:0/469270250 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd4005040 con 0x7fefe00f6510 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.205+0000 7fefde59c700 1 --2- 172.21.15.173:0/469270250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe00f6510 0x7fefe01067e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fefd40089f0 tx=0x7fefd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.205+0000 7fefceffd700 1 -- 172.21.15.173:0/469270250 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefd4009320 con 0x7fefe00f6510 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.205+0000 7fefe4e2e700 1 -- 172.21.15.173:0/469270250 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe0106ed0 con 0x7fefe00f6510 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.205+0000 7fefceffd700 1 -- 172.21.15.173:0/469270250 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefd4013380 con 0x7fefe00f6510 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.205+0000 7fefceffd700 1 -- 172.21.15.173:0/469270250 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefd401ed80 con 0x7fefe00f6510 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.205+0000 7fefe4e2e700 1 -- 172.21.15.173:0/469270250 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe0107360 con 0x7fefe00f6510 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.206+0000 7fefceffd700 1 -- 172.21.15.173:0/469270250 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefd401e420 con 0x7fefe00f6510 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.206+0000 7fefe4e2e700 1 -- 172.21.15.173:0/469270250 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe0045ad0 con 0x7fefe00f6510 2024-02-19T21:59:19.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.206+0000 7fefceffd700 1 --2- 172.21.15.173:0/469270250 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fefc802ff60 0x7fefc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:19.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.206+0000 7fefceffd700 1 -- 172.21.15.173:0/469270250 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefd4040ff0 con 0x7fefe00f6510 2024-02-19T21:59:19.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.206+0000 7fefddd9b700 1 --2- 172.21.15.173:0/469270250 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fefc802ff60 0x7fefc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:19.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.206+0000 7fefddd9b700 1 --2- 172.21.15.173:0/469270250 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fefc802ff60 0x7fefc8032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fefd00096a0 tx=0x7fefd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:19.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.211+0000 7fefceffd700 1 -- 172.21.15.173:0/469270250 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefd40095f0 con 0x7fefe00f6510 2024-02-19T21:59:19.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.417+0000 7fefe4e2e700 1 -- 172.21.15.173:0/469270250 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefe01081b0 con 0x7fefe00f6510 2024-02-19T21:59:19.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.417+0000 7fefceffd700 1 -- 172.21.15.173:0/469270250 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fefd401cb10 con 0x7fefe00f6510 2024-02-19T21:59:19.418 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:19.418 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:19.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.420+0000 7fefe4e2e700 1 -- 172.21.15.173:0/469270250 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fefc802ff60 msgr2=0x7fefc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:19.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.420+0000 7fefe4e2e700 1 --2- 172.21.15.173:0/469270250 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fefc802ff60 0x7fefc8032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fefd00096a0 tx=0x7fefd0006b40).stop 2024-02-19T21:59:19.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.420+0000 7fefe4e2e700 1 -- 172.21.15.173:0/469270250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe00f6510 msgr2=0x7fefe01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:19.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.420+0000 7fefe4e2e700 1 --2- 172.21.15.173:0/469270250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe00f6510 0x7fefe01067e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fefd40089f0 tx=0x7fefd400b040).stop 2024-02-19T21:59:19.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.420+0000 7fefe4e2e700 1 -- 172.21.15.173:0/469270250 shutdown_connections 2024-02-19T21:59:19.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.420+0000 7fefe4e2e700 1 --2- 172.21.15.173:0/469270250 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fefc802ff60 0x7fefc8032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:19.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.420+0000 7fefe4e2e700 1 --2- 172.21.15.173:0/469270250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe00f6510 0x7fefe01067e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:19.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.420+0000 7fefe4e2e700 1 -- 172.21.15.173:0/469270250 >> 172.21.15.173:0/469270250 conn(0x7fefe00f1c70 msgr2=0x7fefe00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:19.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.420+0000 7fefe4e2e700 1 -- 172.21.15.173:0/469270250 shutdown_connections 2024-02-19T21:59:19.422 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:19.420+0000 7fefe4e2e700 1 -- 172.21.15.173:0/469270250 wait complete. 2024-02-19T21:59:19.422 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:19.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:19 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:19.418534+0000 mon.smithi143 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.173:0/469270250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:18.483638+0000 mgr.smithi143.bissbc (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:21.084 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:21.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:20.565962+0000 mon.smithi143 (mon.0) 194 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:59:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:20.566182+0000 mon.smithi143 (mon.0) 195 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T21:59:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:20.483994+0000 mgr.smithi143.bissbc (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:22.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.946+0000 7feb880ed700 1 -- 172.21.15.173:0/134305047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb800f6510 msgr2=0x7feb800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:22.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.946+0000 7feb880ed700 1 --2- 172.21.15.173:0/134305047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb800f6510 0x7feb800f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7feb74009fb0 tx=0x7feb74009b30).stop 2024-02-19T21:59:22.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.946+0000 7feb880ed700 1 -- 172.21.15.173:0/134305047 shutdown_connections 2024-02-19T21:59:22.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.946+0000 7feb880ed700 1 --2- 172.21.15.173:0/134305047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb800f6510 0x7feb800f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:22.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.946+0000 7feb880ed700 1 -- 172.21.15.173:0/134305047 >> 172.21.15.173:0/134305047 conn(0x7feb800f1c70 msgr2=0x7feb800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:22.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.946+0000 7feb880ed700 1 -- 172.21.15.173:0/134305047 shutdown_connections 2024-02-19T21:59:22.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.946+0000 7feb880ed700 1 -- 172.21.15.173:0/134305047 wait complete. 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.947+0000 7feb880ed700 1 Processor -- start 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.947+0000 7feb880ed700 1 -- start start 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.947+0000 7feb880ed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb800f6510 0x7feb801067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.947+0000 7feb880ed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb80106cd0 con 0x7feb800f6510 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.947+0000 7feb85e89700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb800f6510 0x7feb801067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.947+0000 7feb85e89700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb800f6510 0x7feb801067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:54140/0 (socket says 172.21.15.173:54140) 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.947+0000 7feb85e89700 1 -- 172.21.15.173:0/1669071866 learned_addr learned my addr 172.21.15.173:0/1669071866 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.947+0000 7feb85e89700 1 -- 172.21.15.173:0/1669071866 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb74005040 con 0x7feb800f6510 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.948+0000 7feb85e89700 1 --2- 172.21.15.173:0/1669071866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb800f6510 0x7feb801067e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7feb74005010 tx=0x7feb7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.948+0000 7feb727fc700 1 -- 172.21.15.173:0/1669071866 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb74009320 con 0x7feb800f6510 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.948+0000 7feb880ed700 1 -- 172.21.15.173:0/1669071866 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb80106ed0 con 0x7feb800f6510 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.948+0000 7feb727fc700 1 -- 172.21.15.173:0/1669071866 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb74013380 con 0x7feb800f6510 2024-02-19T21:59:22.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.948+0000 7feb727fc700 1 -- 172.21.15.173:0/1669071866 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb74014e50 con 0x7feb800f6510 2024-02-19T21:59:22.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.948+0000 7feb880ed700 1 -- 172.21.15.173:0/1669071866 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb80107360 con 0x7feb800f6510 2024-02-19T21:59:22.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.949+0000 7feb727fc700 1 -- 172.21.15.173:0/1669071866 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb740263a0 con 0x7feb800f6510 2024-02-19T21:59:22.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.949+0000 7feb727fc700 1 --2- 172.21.15.173:0/1669071866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb6c02ff10 0x7feb6c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:22.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.949+0000 7feb727fc700 1 -- 172.21.15.173:0/1669071866 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb74040ec0 con 0x7feb800f6510 2024-02-19T21:59:22.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.949+0000 7feb880ed700 1 -- 172.21.15.173:0/1669071866 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb80045ad0 con 0x7feb800f6510 2024-02-19T21:59:22.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.949+0000 7feb85688700 1 --2- 172.21.15.173:0/1669071866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb6c02ff10 0x7feb6c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:22.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.949+0000 7feb85688700 1 --2- 172.21.15.173:0/1669071866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb6c02ff10 0x7feb6c0323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7feb7c0096a0 tx=0x7feb7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:22.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:22.954+0000 7feb727fc700 1 -- 172.21.15.173:0/1669071866 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb74012920 con 0x7feb800f6510 2024-02-19T21:59:23.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.158+0000 7feb880ed700 1 -- 172.21.15.173:0/1669071866 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb80108080 con 0x7feb800f6510 2024-02-19T21:59:23.158 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.158+0000 7feb727fc700 1 -- 172.21.15.173:0/1669071866 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feb74012330 con 0x7feb800f6510 2024-02-19T21:59:23.158 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:23.158 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:23.159 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.160+0000 7feb880ed700 1 -- 172.21.15.173:0/1669071866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb6c02ff10 msgr2=0x7feb6c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:23.159 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.160+0000 7feb880ed700 1 --2- 172.21.15.173:0/1669071866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb6c02ff10 0x7feb6c0323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7feb7c0096a0 tx=0x7feb7c006b40).stop 2024-02-19T21:59:23.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.160+0000 7feb880ed700 1 -- 172.21.15.173:0/1669071866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb800f6510 msgr2=0x7feb801067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:23.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.160+0000 7feb880ed700 1 --2- 172.21.15.173:0/1669071866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb800f6510 0x7feb801067e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7feb74005010 tx=0x7feb7400b040).stop 2024-02-19T21:59:23.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.160+0000 7feb880ed700 1 -- 172.21.15.173:0/1669071866 shutdown_connections 2024-02-19T21:59:23.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.160+0000 7feb880ed700 1 --2- 172.21.15.173:0/1669071866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb6c02ff10 0x7feb6c0323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:23.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.160+0000 7feb880ed700 1 --2- 172.21.15.173:0/1669071866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb800f6510 0x7feb801067e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:23.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.160+0000 7feb880ed700 1 -- 172.21.15.173:0/1669071866 >> 172.21.15.173:0/1669071866 conn(0x7feb800f1c70 msgr2=0x7feb800f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:23.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.160+0000 7feb880ed700 1 -- 172.21.15.173:0/1669071866 shutdown_connections 2024-02-19T21:59:23.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:23.160+0000 7feb880ed700 1 -- 172.21.15.173:0/1669071866 wait complete. 2024-02-19T21:59:23.160 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:23.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:23.159422+0000 mon.smithi143 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.173:0/1669071866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:24.827 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:24.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:24.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:22.484268+0000 mgr.smithi143.bissbc (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:26.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.792+0000 7fdd92dd6700 1 -- 172.21.15.173:0/2605395852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd8c0f5410 msgr2=0x7fdd8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:26.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.792+0000 7fdd92dd6700 1 --2- 172.21.15.173:0/2605395852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd8c0f5410 0x7fdd8c0f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fdd80009fb0 tx=0x7fdd80009b30).stop 2024-02-19T21:59:26.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.792+0000 7fdd92dd6700 1 -- 172.21.15.173:0/2605395852 shutdown_connections 2024-02-19T21:59:26.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.792+0000 7fdd92dd6700 1 --2- 172.21.15.173:0/2605395852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd8c0f5410 0x7fdd8c0f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:26.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.792+0000 7fdd92dd6700 1 -- 172.21.15.173:0/2605395852 >> 172.21.15.173:0/2605395852 conn(0x7fdd8c0f0b70 msgr2=0x7fdd8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:26.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.792+0000 7fdd92dd6700 1 -- 172.21.15.173:0/2605395852 shutdown_connections 2024-02-19T21:59:26.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.792+0000 7fdd92dd6700 1 -- 172.21.15.173:0/2605395852 wait complete. 2024-02-19T21:59:26.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.793+0000 7fdd92dd6700 1 Processor -- start 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.793+0000 7fdd92dd6700 1 -- start start 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.793+0000 7fdd92dd6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd8c0f5410 0x7fdd8c109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.793+0000 7fdd92dd6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c060240 con 0x7fdd8c0f5410 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.793+0000 7fdd90b72700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd8c0f5410 0x7fdd8c109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.793+0000 7fdd90b72700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd8c0f5410 0x7fdd8c109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:54146/0 (socket says 172.21.15.173:54146) 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.793+0000 7fdd90b72700 1 -- 172.21.15.173:0/4046727982 learned_addr learned my addr 172.21.15.173:0/4046727982 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.794+0000 7fdd90b72700 1 -- 172.21.15.173:0/4046727982 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd80005040 con 0x7fdd8c0f5410 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.794+0000 7fdd90b72700 1 --2- 172.21.15.173:0/4046727982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd8c0f5410 0x7fdd8c109520 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fdd80005010 tx=0x7fdd8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.794+0000 7fdd897fa700 1 -- 172.21.15.173:0/4046727982 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd80009320 con 0x7fdd8c0f5410 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.794+0000 7fdd92dd6700 1 -- 172.21.15.173:0/4046727982 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8c060440 con 0x7fdd8c0f5410 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.794+0000 7fdd897fa700 1 -- 172.21.15.173:0/4046727982 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd80013380 con 0x7fdd8c0f5410 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.794+0000 7fdd897fa700 1 -- 172.21.15.173:0/4046727982 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd80014e50 con 0x7fdd8c0f5410 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.794+0000 7fdd92dd6700 1 -- 172.21.15.173:0/4046727982 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8c109bd0 con 0x7fdd8c0f5410 2024-02-19T21:59:26.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.795+0000 7fdd897fa700 1 -- 172.21.15.173:0/4046727982 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd80014420 con 0x7fdd8c0f5410 2024-02-19T21:59:26.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.795+0000 7fdd92dd6700 1 -- 172.21.15.173:0/4046727982 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8c045ad0 con 0x7fdd8c0f5410 2024-02-19T21:59:26.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.795+0000 7fdd897fa700 1 --2- 172.21.15.173:0/4046727982 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdd7c02ff10 0x7fdd7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:26.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.795+0000 7fdd897fa700 1 -- 172.21.15.173:0/4046727982 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd80040e30 con 0x7fdd8c0f5410 2024-02-19T21:59:26.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.795+0000 7fdd8bfff700 1 --2- 172.21.15.173:0/4046727982 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdd7c02ff10 0x7fdd7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:26.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.796+0000 7fdd8bfff700 1 --2- 172.21.15.173:0/4046727982 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdd7c02ff10 0x7fdd7c0323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fdd780096a0 tx=0x7fdd78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:26.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:26.800+0000 7fdd897fa700 1 -- 172.21.15.173:0/4046727982 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd800095f0 con 0x7fdd8c0f5410 2024-02-19T21:59:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:24.484556+0000 mgr.smithi143.bissbc (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:27.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.003+0000 7fdd92dd6700 1 -- 172.21.15.173:0/4046727982 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd8c10a3f0 con 0x7fdd8c0f5410 2024-02-19T21:59:27.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.004+0000 7fdd897fa700 1 -- 172.21.15.173:0/4046727982 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd8001db70 con 0x7fdd8c0f5410 2024-02-19T21:59:27.004 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:27.004 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:27.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.006+0000 7fdd92dd6700 1 -- 172.21.15.173:0/4046727982 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdd7c02ff10 msgr2=0x7fdd7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:27.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.006+0000 7fdd92dd6700 1 --2- 172.21.15.173:0/4046727982 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdd7c02ff10 0x7fdd7c0323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fdd780096a0 tx=0x7fdd78006b40).stop 2024-02-19T21:59:27.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.006+0000 7fdd92dd6700 1 -- 172.21.15.173:0/4046727982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd8c0f5410 msgr2=0x7fdd8c109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:27.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.006+0000 7fdd92dd6700 1 --2- 172.21.15.173:0/4046727982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd8c0f5410 0x7fdd8c109520 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fdd80005010 tx=0x7fdd8000b040).stop 2024-02-19T21:59:27.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.006+0000 7fdd92dd6700 1 -- 172.21.15.173:0/4046727982 shutdown_connections 2024-02-19T21:59:27.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.006+0000 7fdd92dd6700 1 --2- 172.21.15.173:0/4046727982 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdd7c02ff10 0x7fdd7c0323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:27.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.006+0000 7fdd92dd6700 1 --2- 172.21.15.173:0/4046727982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd8c0f5410 0x7fdd8c109520 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:27.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.006+0000 7fdd92dd6700 1 -- 172.21.15.173:0/4046727982 >> 172.21.15.173:0/4046727982 conn(0x7fdd8c0f0b70 msgr2=0x7fdd8c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:27.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.006+0000 7fdd92dd6700 1 -- 172.21.15.173:0/4046727982 shutdown_connections 2024-02-19T21:59:27.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:27.006+0000 7fdd92dd6700 1 -- 172.21.15.173:0/4046727982 wait complete. 2024-02-19T21:59:27.007 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:27.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:27 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:27.005191+0000 mon.smithi143 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.173:0/4046727982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:28.708 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:28.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:26.484821+0000 mgr.smithi143.bissbc (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:30.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.474+0000 7f6778c6f700 1 -- 172.21.15.173:0/519063596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67740f51a0 msgr2=0x7f67740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:30.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.474+0000 7f6778c6f700 1 --2- 172.21.15.173:0/519063596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67740f51a0 0x7f67740f55c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f6764009fb0 tx=0x7f6764009b30).stop 2024-02-19T21:59:30.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.475+0000 7f6778c6f700 1 -- 172.21.15.173:0/519063596 shutdown_connections 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.475+0000 7f6778c6f700 1 --2- 172.21.15.173:0/519063596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67740f51a0 0x7f67740f55c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.475+0000 7f6778c6f700 1 -- 172.21.15.173:0/519063596 >> 172.21.15.173:0/519063596 conn(0x7f67740f09a0 msgr2=0x7f67740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.475+0000 7f6778c6f700 1 -- 172.21.15.173:0/519063596 shutdown_connections 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.475+0000 7f6778c6f700 1 -- 172.21.15.173:0/519063596 wait complete. 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.475+0000 7f6778c6f700 1 Processor -- start 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.475+0000 7f6778c6f700 1 -- start start 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.475+0000 7f6778c6f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67740f51a0 0x7f67740f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.475+0000 7f6778c6f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67740f13e0 con 0x7f67740f51a0 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.476+0000 7f67737fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67740f51a0 0x7f67740f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.476+0000 7f67737fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67740f51a0 0x7f67740f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:54168/0 (socket says 172.21.15.173:54168) 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.476+0000 7f67737fe700 1 -- 172.21.15.173:0/299014208 learned_addr learned my addr 172.21.15.173:0/299014208 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.476+0000 7f67737fe700 1 -- 172.21.15.173:0/299014208 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6764005040 con 0x7f67740f51a0 2024-02-19T21:59:30.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.476+0000 7f67737fe700 1 --2- 172.21.15.173:0/299014208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67740f51a0 0x7f67740f2f60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f67640089f0 tx=0x7f676400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:30.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.476+0000 7f675bfff700 1 -- 172.21.15.173:0/299014208 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6764009320 con 0x7f67740f51a0 2024-02-19T21:59:30.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.476+0000 7f6778c6f700 1 -- 172.21.15.173:0/299014208 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67740f1580 con 0x7f67740f51a0 2024-02-19T21:59:30.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.476+0000 7f675bfff700 1 -- 172.21.15.173:0/299014208 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6764013380 con 0x7f67740f51a0 2024-02-19T21:59:30.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.476+0000 7f675bfff700 1 -- 172.21.15.173:0/299014208 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6764014e20 con 0x7f67740f51a0 2024-02-19T21:59:30.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.476+0000 7f6778c6f700 1 -- 172.21.15.173:0/299014208 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67740f1a10 con 0x7f67740f51a0 2024-02-19T21:59:30.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.477+0000 7f675bfff700 1 -- 172.21.15.173:0/299014208 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6764014420 con 0x7f67740f51a0 2024-02-19T21:59:30.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.477+0000 7f6778c6f700 1 -- 172.21.15.173:0/299014208 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6774059100 con 0x7f67740f51a0 2024-02-19T21:59:30.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.477+0000 7f675bfff700 1 --2- 172.21.15.173:0/299014208 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f675c02ff10 0x7f675c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:30.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.477+0000 7f675bfff700 1 -- 172.21.15.173:0/299014208 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6764040f90 con 0x7f67740f51a0 2024-02-19T21:59:30.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.477+0000 7f6772ffd700 1 --2- 172.21.15.173:0/299014208 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f675c02ff10 0x7f675c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:30.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.478+0000 7f6772ffd700 1 --2- 172.21.15.173:0/299014208 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f675c02ff10 0x7f675c0323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f67680096a0 tx=0x7f6768006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:30.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.482+0000 7f675bfff700 1 -- 172.21.15.173:0/299014208 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f676401d7f0 con 0x7f67740f51a0 2024-02-19T21:59:30.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.685+0000 7f6778c6f700 1 -- 172.21.15.173:0/299014208 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6774045ad0 con 0x7f67740f51a0 2024-02-19T21:59:30.686 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.686+0000 7f675bfff700 1 -- 172.21.15.173:0/299014208 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f676401d7f0 con 0x7f67740f51a0 2024-02-19T21:59:30.686 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:30.686 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:30.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.688+0000 7f6778c6f700 1 -- 172.21.15.173:0/299014208 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f675c02ff10 msgr2=0x7f675c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:30.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.688+0000 7f6778c6f700 1 --2- 172.21.15.173:0/299014208 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f675c02ff10 0x7f675c0323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f67680096a0 tx=0x7f6768006b40).stop 2024-02-19T21:59:30.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.688+0000 7f6778c6f700 1 -- 172.21.15.173:0/299014208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67740f51a0 msgr2=0x7f67740f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:30.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.688+0000 7f6778c6f700 1 --2- 172.21.15.173:0/299014208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67740f51a0 0x7f67740f2f60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f67640089f0 tx=0x7f676400b040).stop 2024-02-19T21:59:30.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.688+0000 7f6778c6f700 1 -- 172.21.15.173:0/299014208 shutdown_connections 2024-02-19T21:59:30.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.688+0000 7f6778c6f700 1 --2- 172.21.15.173:0/299014208 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f675c02ff10 0x7f675c0323d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:30.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.688+0000 7f6778c6f700 1 --2- 172.21.15.173:0/299014208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67740f51a0 0x7f67740f2f60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:30.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.688+0000 7f6778c6f700 1 -- 172.21.15.173:0/299014208 >> 172.21.15.173:0/299014208 conn(0x7f67740f09a0 msgr2=0x7f677405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:30.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.688+0000 7f6778c6f700 1 -- 172.21.15.173:0/299014208 shutdown_connections 2024-02-19T21:59:30.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:30.688+0000 7f6778c6f700 1 -- 172.21.15.173:0/299014208 wait complete. 2024-02-19T21:59:30.689 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:28.485027+0000 mgr.smithi143.bissbc (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:31.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:31 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:30.687447+0000 mon.smithi143 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.173:0/299014208' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:32.376 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:32.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:32.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:30.485242+0000 mgr.smithi143.bissbc (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:34.297 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.297+0000 7f097623b700 1 -- 172.21.15.173:0/1564936865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09700601c0 msgr2=0x7f09700f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:34.297 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.297+0000 7f097623b700 1 --2- 172.21.15.173:0/1564936865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09700601c0 0x7f09700f91b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f0964009fb0 tx=0x7f0964009b30).stop 2024-02-19T21:59:34.297 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.297+0000 7f097623b700 1 -- 172.21.15.173:0/1564936865 shutdown_connections 2024-02-19T21:59:34.297 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.297+0000 7f097623b700 1 --2- 172.21.15.173:0/1564936865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09700601c0 0x7f09700f91b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:34.297 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.297+0000 7f097623b700 1 -- 172.21.15.173:0/1564936865 >> 172.21.15.173:0/1564936865 conn(0x7f09700f0b70 msgr2=0x7f09700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:34.297 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.297+0000 7f097623b700 1 -- 172.21.15.173:0/1564936865 shutdown_connections 2024-02-19T21:59:34.297 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.297+0000 7f097623b700 1 -- 172.21.15.173:0/1564936865 wait complete. 2024-02-19T21:59:34.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.298+0000 7f097623b700 1 Processor -- start 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.298+0000 7f097623b700 1 -- start start 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.298+0000 7f097623b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09700601c0 0x7f09701067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.298+0000 7f097623b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0970106cb0 con 0x7f09700601c0 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.298+0000 7f096f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09700601c0 0x7f09701067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.298+0000 7f096f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09700601c0 0x7f09701067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:39716/0 (socket says 172.21.15.173:39716) 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.298+0000 7f096f7fe700 1 -- 172.21.15.173:0/3349168873 learned_addr learned my addr 172.21.15.173:0/3349168873 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.298+0000 7f096f7fe700 1 -- 172.21.15.173:0/3349168873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0964005040 con 0x7f09700601c0 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.299+0000 7f096f7fe700 1 --2- 172.21.15.173:0/3349168873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09700601c0 0x7f09701067c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f09640089f0 tx=0x7f096400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.299+0000 7f0957fff700 1 -- 172.21.15.173:0/3349168873 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0964009320 con 0x7f09700601c0 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.299+0000 7f0957fff700 1 -- 172.21.15.173:0/3349168873 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0964013380 con 0x7f09700601c0 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.299+0000 7f097623b700 1 -- 172.21.15.173:0/3349168873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0970106eb0 con 0x7f09700601c0 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.299+0000 7f0957fff700 1 -- 172.21.15.173:0/3349168873 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0964014e20 con 0x7f09700601c0 2024-02-19T21:59:34.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.299+0000 7f097623b700 1 -- 172.21.15.173:0/3349168873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0970107340 con 0x7f09700601c0 2024-02-19T21:59:34.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.299+0000 7f0957fff700 1 -- 172.21.15.173:0/3349168873 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09640263a0 con 0x7f09700601c0 2024-02-19T21:59:34.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.300+0000 7f0957fff700 1 --2- 172.21.15.173:0/3349168873 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f095802ff10 0x7f09580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:34.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.300+0000 7f097623b700 1 -- 172.21.15.173:0/3349168873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0970045ad0 con 0x7f09700601c0 2024-02-19T21:59:34.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.300+0000 7f0957fff700 1 -- 172.21.15.173:0/3349168873 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0964040f90 con 0x7f09700601c0 2024-02-19T21:59:34.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.300+0000 7f096effd700 1 --2- 172.21.15.173:0/3349168873 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f095802ff10 0x7f09580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:34.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.300+0000 7f096effd700 1 --2- 172.21.15.173:0/3349168873 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f095802ff10 0x7f09580323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f09600096a0 tx=0x7f0960006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:34.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.304+0000 7f0957fff700 1 -- 172.21.15.173:0/3349168873 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0964012920 con 0x7f09700601c0 2024-02-19T21:59:34.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.510+0000 7f097623b700 1 -- 172.21.15.173:0/3349168873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09701081f0 con 0x7f09700601c0 2024-02-19T21:59:34.510 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.511+0000 7f0957fff700 1 -- 172.21.15.173:0/3349168873 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f096402b020 con 0x7f09700601c0 2024-02-19T21:59:34.511 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:34.511 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:34.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.512+0000 7f097623b700 1 -- 172.21.15.173:0/3349168873 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f095802ff10 msgr2=0x7f09580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:34.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.512+0000 7f097623b700 1 --2- 172.21.15.173:0/3349168873 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f095802ff10 0x7f09580323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f09600096a0 tx=0x7f0960006b40).stop 2024-02-19T21:59:34.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.512+0000 7f097623b700 1 -- 172.21.15.173:0/3349168873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09700601c0 msgr2=0x7f09701067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:34.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.512+0000 7f097623b700 1 --2- 172.21.15.173:0/3349168873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09700601c0 0x7f09701067c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f09640089f0 tx=0x7f096400b040).stop 2024-02-19T21:59:34.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.512+0000 7f097623b700 1 -- 172.21.15.173:0/3349168873 shutdown_connections 2024-02-19T21:59:34.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.512+0000 7f097623b700 1 --2- 172.21.15.173:0/3349168873 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f095802ff10 0x7f09580323d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:34.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.512+0000 7f097623b700 1 --2- 172.21.15.173:0/3349168873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09700601c0 0x7f09701067c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:34.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.512+0000 7f097623b700 1 -- 172.21.15.173:0/3349168873 >> 172.21.15.173:0/3349168873 conn(0x7f09700f0b70 msgr2=0x7f09700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:34.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.512+0000 7f097623b700 1 -- 172.21.15.173:0/3349168873 shutdown_connections 2024-02-19T21:59:34.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:34.513+0000 7f097623b700 1 -- 172.21.15.173:0/3349168873 wait complete. 2024-02-19T21:59:34.513 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:32.485496+0000 mgr.smithi143.bissbc (mgr.14162) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:33.499649+0000 mon.smithi143 (mon.0) 199 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi143", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T21:59:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:33.502265+0000 mon.smithi143 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi143", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T21:59:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:33.503069+0000 mon.smithi143 (mon.0) 201 : audit [DBG] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:59:35.799 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:59:33.498015+0000 mgr.smithi143.bissbc (mgr.14162) 158 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi143 ... 2024-02-19T21:59:35.799 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T21:59:35.799 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T21:59:35.799 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:55:12Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:55:49Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:56:26Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:57:41Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:58:18Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:58:55Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Traceback (most recent call last): 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T21:59:35.800 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: main() 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: r = ctx.func(ctx) 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: return func(ctx) 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: raise RuntimeError('uid/gid not found') 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: RuntimeError: uid/gid not found 2024-02-19T21:59:35.801 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Traceback (most recent call last): 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: yield (conn, connr) 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: code, '\n'.join(err))) 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi143 ... 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:55:12Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:55:49Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:56:26Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T21:59:35.802 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:57:41Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:58:18Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T21:58:55Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Traceback (most recent call last): 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: main() 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: r = ctx.func(ctx) 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: return func(ctx) 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T21:59:35.803 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T21:59:35.804 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: raise RuntimeError('uid/gid not found') 2024-02-19T21:59:35.804 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: RuntimeError: uid/gid not found 2024-02-19T21:59:35.804 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:59:33.503815+0000 mgr.smithi143.bissbc (mgr.14162) 159 : cephadm [INF] Deploying daemon crash.smithi143 on smithi143 2024-02-19T21:59:35.804 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:34.512124+0000 mon.smithi143 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.173:0/3349168873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:36.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:36.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:34.485762+0000 mgr.smithi143.bissbc (mgr.14162) 160 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:35.954302+0000 mon.smithi143 (mon.0) 203 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:59:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:36.018570+0000 mon.smithi143 (mon.0) 204 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:59:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:36.020684+0000 mon.smithi143 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:59:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:36.021137+0000 mon.smithi143 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T21:59:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:36.023806+0000 mon.smithi143 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:59:37.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:37 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:36.021565+0000 mgr.smithi143.bissbc (mgr.14162) 161 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T21:59:37.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:37 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:59:36.026780+0000 mgr.smithi143.bissbc (mgr.14162) 162 : cephadm [INF] Deploying daemon grafana.smithi143 on smithi143 2024-02-19T21:59:38.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.126+0000 7f660ba2e700 1 -- 172.21.15.173:0/2670742481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66040f5410 msgr2=0x7f66040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:38.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.126+0000 7f660ba2e700 1 --2- 172.21.15.173:0/2670742481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66040f5410 0x7f66040f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f65f8009fb0 tx=0x7f65f8009b30).stop 2024-02-19T21:59:38.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.126+0000 7f660ba2e700 1 -- 172.21.15.173:0/2670742481 shutdown_connections 2024-02-19T21:59:38.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.126+0000 7f660ba2e700 1 --2- 172.21.15.173:0/2670742481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66040f5410 0x7f66040f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:38.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.126+0000 7f660ba2e700 1 -- 172.21.15.173:0/2670742481 >> 172.21.15.173:0/2670742481 conn(0x7f66040f0b70 msgr2=0x7f66040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:38.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.126+0000 7f660ba2e700 1 -- 172.21.15.173:0/2670742481 shutdown_connections 2024-02-19T21:59:38.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.126+0000 7f660ba2e700 1 -- 172.21.15.173:0/2670742481 wait complete. 2024-02-19T21:59:38.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.127+0000 7f660ba2e700 1 Processor -- start 2024-02-19T21:59:38.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.127+0000 7f660ba2e700 1 -- start start 2024-02-19T21:59:38.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.127+0000 7f660ba2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66040f5410 0x7f6604109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.127+0000 7f660ba2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6604060220 con 0x7f66040f5410 2024-02-19T21:59:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.127+0000 7f66097ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66040f5410 0x7f6604109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.127+0000 7f66097ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66040f5410 0x7f6604109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:39736/0 (socket says 172.21.15.173:39736) 2024-02-19T21:59:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.127+0000 7f66097ca700 1 -- 172.21.15.173:0/2743395107 learned_addr learned my addr 172.21.15.173:0/2743395107 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.127+0000 7f66097ca700 1 -- 172.21.15.173:0/2743395107 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f8005040 con 0x7f66040f5410 2024-02-19T21:59:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.127+0000 7f66097ca700 1 --2- 172.21.15.173:0/2743395107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66040f5410 0x7f6604109500 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f65f8005010 tx=0x7f65f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.128+0000 7f65f5ffb700 1 -- 172.21.15.173:0/2743395107 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65f8009320 con 0x7f66040f5410 2024-02-19T21:59:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.128+0000 7f65f5ffb700 1 -- 172.21.15.173:0/2743395107 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65f8013380 con 0x7f66040f5410 2024-02-19T21:59:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.128+0000 7f660ba2e700 1 -- 172.21.15.173:0/2743395107 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6604060420 con 0x7f66040f5410 2024-02-19T21:59:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.128+0000 7f65f5ffb700 1 -- 172.21.15.173:0/2743395107 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65f8014e50 con 0x7f66040f5410 2024-02-19T21:59:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.128+0000 7f660ba2e700 1 -- 172.21.15.173:0/2743395107 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6604109bb0 con 0x7f66040f5410 2024-02-19T21:59:38.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.129+0000 7f65f5ffb700 1 -- 172.21.15.173:0/2743395107 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65f80095f0 con 0x7f66040f5410 2024-02-19T21:59:38.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.129+0000 7f660ba2e700 1 -- 172.21.15.173:0/2743395107 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6604059100 con 0x7f66040f5410 2024-02-19T21:59:38.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.129+0000 7f65f5ffb700 1 --2- 172.21.15.173:0/2743395107 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f65f002ff10 0x7f65f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:38.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.129+0000 7f65f5ffb700 1 -- 172.21.15.173:0/2743395107 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f65f8040f80 con 0x7f66040f5410 2024-02-19T21:59:38.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.129+0000 7f6608fc9700 1 --2- 172.21.15.173:0/2743395107 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f65f002ff10 0x7f65f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:38.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.130+0000 7f6608fc9700 1 --2- 172.21.15.173:0/2743395107 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f65f002ff10 0x7f65f00323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f66000096a0 tx=0x7f6600006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:38.134 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.134+0000 7f65f5ffb700 1 -- 172.21.15.173:0/2743395107 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65f8014420 con 0x7f66040f5410 2024-02-19T21:59:38.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.339+0000 7f660ba2e700 1 -- 172.21.15.173:0/2743395107 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66040fafa0 con 0x7f66040f5410 2024-02-19T21:59:38.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.340+0000 7f65f5ffb700 1 -- 172.21.15.173:0/2743395107 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f65f8010230 con 0x7f66040f5410 2024-02-19T21:59:38.340 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:38.340 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:38.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.341+0000 7f660ba2e700 1 -- 172.21.15.173:0/2743395107 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f65f002ff10 msgr2=0x7f65f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:38.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.341+0000 7f660ba2e700 1 --2- 172.21.15.173:0/2743395107 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f65f002ff10 0x7f65f00323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f66000096a0 tx=0x7f6600006b40).stop 2024-02-19T21:59:38.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.341+0000 7f660ba2e700 1 -- 172.21.15.173:0/2743395107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66040f5410 msgr2=0x7f6604109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:38.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.341+0000 7f660ba2e700 1 --2- 172.21.15.173:0/2743395107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66040f5410 0x7f6604109500 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f65f8005010 tx=0x7f65f800b040).stop 2024-02-19T21:59:38.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.342+0000 7f660ba2e700 1 -- 172.21.15.173:0/2743395107 shutdown_connections 2024-02-19T21:59:38.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.342+0000 7f660ba2e700 1 --2- 172.21.15.173:0/2743395107 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f65f002ff10 0x7f65f00323d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:38.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.342+0000 7f660ba2e700 1 --2- 172.21.15.173:0/2743395107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66040f5410 0x7f6604109500 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:38.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.342+0000 7f660ba2e700 1 -- 172.21.15.173:0/2743395107 >> 172.21.15.173:0/2743395107 conn(0x7f66040f0b70 msgr2=0x7f66040f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:38.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.342+0000 7f660ba2e700 1 -- 172.21.15.173:0/2743395107 shutdown_connections 2024-02-19T21:59:38.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:38.342+0000 7f660ba2e700 1 -- 172.21.15.173:0/2743395107 wait complete. 2024-02-19T21:59:38.342 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:36.486025+0000 mgr.smithi143.bissbc (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:38.341173+0000 mon.smithi143 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.173:0/2743395107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:40.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:40.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:38.486384+0000 mgr.smithi143.bissbc (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:42.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.179+0000 7f9eb266d700 1 -- 172.21.15.173:0/2956160719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eac0f6510 msgr2=0x7f9eac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:42.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.179+0000 7f9eb266d700 1 --2- 172.21.15.173:0/2956160719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eac0f6510 0x7f9eac0f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f9e9c009fb0 tx=0x7f9e9c009b30).stop 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.179+0000 7f9eb266d700 1 -- 172.21.15.173:0/2956160719 shutdown_connections 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.179+0000 7f9eb266d700 1 --2- 172.21.15.173:0/2956160719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eac0f6510 0x7f9eac0f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.179+0000 7f9eb266d700 1 -- 172.21.15.173:0/2956160719 >> 172.21.15.173:0/2956160719 conn(0x7f9eac0f1c70 msgr2=0x7f9eac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.179+0000 7f9eb266d700 1 -- 172.21.15.173:0/2956160719 shutdown_connections 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.179+0000 7f9eb266d700 1 -- 172.21.15.173:0/2956160719 wait complete. 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.179+0000 7f9eb266d700 1 Processor -- start 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.179+0000 7f9eb266d700 1 -- start start 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.180+0000 7f9eb266d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eac0f6510 0x7f9eac06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.180+0000 7f9eb266d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac069e20 con 0x7f9eac0f6510 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.180+0000 7f9eabfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eac0f6510 0x7f9eac06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.180+0000 7f9eabfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eac0f6510 0x7f9eac06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:51598/0 (socket says 172.21.15.173:51598) 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.180+0000 7f9eabfff700 1 -- 172.21.15.173:0/4212979757 learned_addr learned my addr 172.21.15.173:0/4212979757 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.180+0000 7f9eabfff700 1 -- 172.21.15.173:0/4212979757 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c005040 con 0x7f9eac0f6510 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.180+0000 7f9eabfff700 1 --2- 172.21.15.173:0/4212979757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eac0f6510 0x7f9eac06b9a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f9e9c005010 tx=0x7f9e9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:42.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.180+0000 7f9ea97fa700 1 -- 172.21.15.173:0/4212979757 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e9c009320 con 0x7f9eac0f6510 2024-02-19T21:59:42.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.180+0000 7f9ea97fa700 1 -- 172.21.15.173:0/4212979757 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e9c013380 con 0x7f9eac0f6510 2024-02-19T21:59:42.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.180+0000 7f9eb266d700 1 -- 172.21.15.173:0/4212979757 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eac069fc0 con 0x7f9eac0f6510 2024-02-19T21:59:42.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.181+0000 7f9ea97fa700 1 -- 172.21.15.173:0/4212979757 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e9c014e50 con 0x7f9eac0f6510 2024-02-19T21:59:42.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.181+0000 7f9eb266d700 1 -- 172.21.15.173:0/4212979757 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eac06a450 con 0x7f9eac0f6510 2024-02-19T21:59:42.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.181+0000 7f9ea97fa700 1 -- 172.21.15.173:0/4212979757 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e9c0095f0 con 0x7f9eac0f6510 2024-02-19T21:59:42.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.181+0000 7f9eb266d700 1 -- 172.21.15.173:0/4212979757 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eac059100 con 0x7f9eac0f6510 2024-02-19T21:59:42.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.181+0000 7f9ea97fa700 1 --2- 172.21.15.173:0/4212979757 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9e9802ff10 0x7f9e980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:42.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.181+0000 7f9ea97fa700 1 -- 172.21.15.173:0/4212979757 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e9c040f80 con 0x7f9eac0f6510 2024-02-19T21:59:42.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.182+0000 7f9ea3fff700 1 --2- 172.21.15.173:0/4212979757 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9e9802ff10 0x7f9e980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:42.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.182+0000 7f9ea3fff700 1 --2- 172.21.15.173:0/4212979757 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9e9802ff10 0x7f9e980323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9e940096a0 tx=0x7f9e94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:42.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.186+0000 7f9ea97fa700 1 -- 172.21.15.173:0/4212979757 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e9c014420 con 0x7f9eac0f6510 2024-02-19T21:59:42.393 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.393+0000 7f9eb266d700 1 -- 172.21.15.173:0/4212979757 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9eac0fc0a0 con 0x7f9eac0f6510 2024-02-19T21:59:42.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.394+0000 7f9ea97fa700 1 -- 172.21.15.173:0/4212979757 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9e9c010230 con 0x7f9eac0f6510 2024-02-19T21:59:42.394 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:42.394 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:42.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.395+0000 7f9eb266d700 1 -- 172.21.15.173:0/4212979757 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9e9802ff10 msgr2=0x7f9e980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:42.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.395+0000 7f9eb266d700 1 --2- 172.21.15.173:0/4212979757 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9e9802ff10 0x7f9e980323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9e940096a0 tx=0x7f9e94006b40).stop 2024-02-19T21:59:42.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.395+0000 7f9eb266d700 1 -- 172.21.15.173:0/4212979757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eac0f6510 msgr2=0x7f9eac06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:42.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.395+0000 7f9eb266d700 1 --2- 172.21.15.173:0/4212979757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eac0f6510 0x7f9eac06b9a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f9e9c005010 tx=0x7f9e9c00b040).stop 2024-02-19T21:59:42.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.395+0000 7f9eb266d700 1 -- 172.21.15.173:0/4212979757 shutdown_connections 2024-02-19T21:59:42.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.395+0000 7f9eb266d700 1 --2- 172.21.15.173:0/4212979757 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9e9802ff10 0x7f9e980323d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:42.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.395+0000 7f9eb266d700 1 --2- 172.21.15.173:0/4212979757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eac0f6510 0x7f9eac06b9a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:42.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.395+0000 7f9eb266d700 1 -- 172.21.15.173:0/4212979757 >> 172.21.15.173:0/4212979757 conn(0x7f9eac0f1c70 msgr2=0x7f9eac0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:42.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.395+0000 7f9eb266d700 1 -- 172.21.15.173:0/4212979757 shutdown_connections 2024-02-19T21:59:42.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:42.395+0000 7f9eb266d700 1 -- 172.21.15.173:0/4212979757 wait complete. 2024-02-19T21:59:42.398 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:40.486684+0000 mgr.smithi143.bissbc (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:42.394914+0000 mon.smithi143 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.173:0/4212979757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:44.134 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:44.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:42.486888+0000 mgr.smithi143.bissbc (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:46.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.144+0000 7f2b4edbf700 1 -- 172.21.15.173:0/4021876463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b480f5410 msgr2=0x7f2b480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:46.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.144+0000 7f2b4edbf700 1 --2- 172.21.15.173:0/4021876463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b480f5410 0x7f2b480f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f2b3c009fb0 tx=0x7f2b3c009b30).stop 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.144+0000 7f2b4edbf700 1 -- 172.21.15.173:0/4021876463 shutdown_connections 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.144+0000 7f2b4edbf700 1 --2- 172.21.15.173:0/4021876463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b480f5410 0x7f2b480f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.144+0000 7f2b4edbf700 1 -- 172.21.15.173:0/4021876463 >> 172.21.15.173:0/4021876463 conn(0x7f2b480f0b70 msgr2=0x7f2b480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.144+0000 7f2b4edbf700 1 -- 172.21.15.173:0/4021876463 shutdown_connections 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.144+0000 7f2b4edbf700 1 -- 172.21.15.173:0/4021876463 wait complete. 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.145+0000 7f2b4edbf700 1 Processor -- start 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.145+0000 7f2b4edbf700 1 -- start start 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b4edbf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b480f5410 0x7f2b48105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b4edbf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b48105c00 con 0x7f2b480f5410 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b4cb5b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b480f5410 0x7f2b48105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b4cb5b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b480f5410 0x7f2b48105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:51620/0 (socket says 172.21.15.173:51620) 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b4cb5b700 1 -- 172.21.15.173:0/782754310 learned_addr learned my addr 172.21.15.173:0/782754310 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:46.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b4cb5b700 1 -- 172.21.15.173:0/782754310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c005040 con 0x7f2b480f5410 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b4cb5b700 1 --2- 172.21.15.173:0/782754310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b480f5410 0x7f2b48105710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2b3c005010 tx=0x7f2b3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b457fa700 1 -- 172.21.15.173:0/782754310 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b3c009320 con 0x7f2b480f5410 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b4edbf700 1 -- 172.21.15.173:0/782754310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b48105e00 con 0x7f2b480f5410 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b457fa700 1 -- 172.21.15.173:0/782754310 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b3c013380 con 0x7f2b480f5410 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b457fa700 1 -- 172.21.15.173:0/782754310 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b3c014e50 con 0x7f2b480f5410 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.146+0000 7f2b4edbf700 1 -- 172.21.15.173:0/782754310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b48106290 con 0x7f2b480f5410 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.147+0000 7f2b457fa700 1 -- 172.21.15.173:0/782754310 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b3c0263a0 con 0x7f2b480f5410 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.147+0000 7f2b457fa700 1 --2- 172.21.15.173:0/782754310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2b3802ff10 0x7f2b380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.147+0000 7f2b457fa700 1 -- 172.21.15.173:0/782754310 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b3c041140 con 0x7f2b480f5410 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.147+0000 7f2b4edbf700 1 -- 172.21.15.173:0/782754310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b48045ad0 con 0x7f2b480f5410 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.148+0000 7f2b47fff700 1 --2- 172.21.15.173:0/782754310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2b3802ff10 0x7f2b380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.148+0000 7f2b47fff700 1 --2- 172.21.15.173:0/782754310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2b3802ff10 0x7f2b380323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f2b340096a0 tx=0x7f2b34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:46.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.152+0000 7f2b457fa700 1 -- 172.21.15.173:0/782754310 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b3c01cb10 con 0x7f2b480f5410 2024-02-19T21:59:46.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.359+0000 7f2b4edbf700 1 -- 172.21.15.173:0/782754310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b48107140 con 0x7f2b480f5410 2024-02-19T21:59:46.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.359+0000 7f2b457fa700 1 -- 172.21.15.173:0/782754310 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2b3c01cb10 con 0x7f2b480f5410 2024-02-19T21:59:46.359 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:46.359 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:46.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.361+0000 7f2b4edbf700 1 -- 172.21.15.173:0/782754310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2b3802ff10 msgr2=0x7f2b380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:46.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.361+0000 7f2b4edbf700 1 --2- 172.21.15.173:0/782754310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2b3802ff10 0x7f2b380323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f2b340096a0 tx=0x7f2b34006b40).stop 2024-02-19T21:59:46.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.361+0000 7f2b4edbf700 1 -- 172.21.15.173:0/782754310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b480f5410 msgr2=0x7f2b48105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:46.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.361+0000 7f2b4edbf700 1 --2- 172.21.15.173:0/782754310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b480f5410 0x7f2b48105710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2b3c005010 tx=0x7f2b3c00b040).stop 2024-02-19T21:59:46.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.361+0000 7f2b4edbf700 1 -- 172.21.15.173:0/782754310 shutdown_connections 2024-02-19T21:59:46.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.361+0000 7f2b4edbf700 1 --2- 172.21.15.173:0/782754310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2b3802ff10 0x7f2b380323d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:46.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.361+0000 7f2b4edbf700 1 --2- 172.21.15.173:0/782754310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b480f5410 0x7f2b48105710 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:46.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.361+0000 7f2b4edbf700 1 -- 172.21.15.173:0/782754310 >> 172.21.15.173:0/782754310 conn(0x7f2b480f0b70 msgr2=0x7f2b480f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:46.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.361+0000 7f2b4edbf700 1 -- 172.21.15.173:0/782754310 shutdown_connections 2024-02-19T21:59:46.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:46.361+0000 7f2b4edbf700 1 -- 172.21.15.173:0/782754310 wait complete. 2024-02-19T21:59:46.362 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:44.487102+0000 mgr.smithi143.bissbc (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:46.360483+0000 mon.smithi143 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.173:0/782754310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:48.046 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:48.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:46.487299+0000 mgr.smithi143.bissbc (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:49.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.856+0000 7f1445965700 1 -- 172.21.15.173:0/3445956442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14400f5410 msgr2=0x7f14400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:49.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.856+0000 7f1445965700 1 --2- 172.21.15.173:0/3445956442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14400f5410 0x7f14400f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f1428009fb0 tx=0x7f1428009b30).stop 2024-02-19T21:59:49.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.856+0000 7f1445965700 1 -- 172.21.15.173:0/3445956442 shutdown_connections 2024-02-19T21:59:49.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.856+0000 7f1445965700 1 --2- 172.21.15.173:0/3445956442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14400f5410 0x7f14400f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:49.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.856+0000 7f1445965700 1 -- 172.21.15.173:0/3445956442 >> 172.21.15.173:0/3445956442 conn(0x7f14400f0b70 msgr2=0x7f14400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:49.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.856+0000 7f1445965700 1 -- 172.21.15.173:0/3445956442 shutdown_connections 2024-02-19T21:59:49.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.856+0000 7f1445965700 1 -- 172.21.15.173:0/3445956442 wait complete. 2024-02-19T21:59:49.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.857+0000 7f1445965700 1 Processor -- start 2024-02-19T21:59:49.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.857+0000 7f1445965700 1 -- start start 2024-02-19T21:59:49.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.857+0000 7f1445965700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14400f5410 0x7f14400fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:49.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.857+0000 7f1445965700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14400fb960 con 0x7f14400f5410 2024-02-19T21:59:49.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.857+0000 7f143effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14400f5410 0x7f14400fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:49.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.857+0000 7f143effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14400f5410 0x7f14400fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:51630/0 (socket says 172.21.15.173:51630) 2024-02-19T21:59:49.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.857+0000 7f143effd700 1 -- 172.21.15.173:0/2873812179 learned_addr learned my addr 172.21.15.173:0/2873812179 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.857+0000 7f143effd700 1 -- 172.21.15.173:0/2873812179 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1428005040 con 0x7f14400f5410 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.858+0000 7f143effd700 1 --2- 172.21.15.173:0/2873812179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14400f5410 0x7f14400fd4e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f1428005010 tx=0x7f142800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.858+0000 7f14377fe700 1 -- 172.21.15.173:0/2873812179 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1428009320 con 0x7f14400f5410 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.858+0000 7f1445965700 1 -- 172.21.15.173:0/2873812179 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14400fbb00 con 0x7f14400f5410 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.858+0000 7f14377fe700 1 -- 172.21.15.173:0/2873812179 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1428013380 con 0x7f14400f5410 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.858+0000 7f14377fe700 1 -- 172.21.15.173:0/2873812179 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1428014e50 con 0x7f14400f5410 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.858+0000 7f1445965700 1 -- 172.21.15.173:0/2873812179 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14400fbf90 con 0x7f14400f5410 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.858+0000 7f14377fe700 1 -- 172.21.15.173:0/2873812179 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14280263a0 con 0x7f14400f5410 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.859+0000 7f1445965700 1 -- 172.21.15.173:0/2873812179 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1440059100 con 0x7f14400f5410 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.859+0000 7f14377fe700 1 --2- 172.21.15.173:0/2873812179 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f142c02ff10 0x7f142c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.859+0000 7f14377fe700 1 -- 172.21.15.173:0/2873812179 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1428040fb0 con 0x7f14400f5410 2024-02-19T21:59:49.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.859+0000 7f143e7fc700 1 --2- 172.21.15.173:0/2873812179 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f142c02ff10 0x7f142c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:49.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.859+0000 7f143e7fc700 1 --2- 172.21.15.173:0/2873812179 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f142c02ff10 0x7f142c0323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f14300096a0 tx=0x7f1430006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:49.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:49.864+0000 7f14377fe700 1 -- 172.21.15.173:0/2873812179 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f142801d470 con 0x7f14400f5410 2024-02-19T21:59:50.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.070+0000 7f1445965700 1 -- 172.21.15.173:0/2873812179 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14400fafa0 con 0x7f14400f5410 2024-02-19T21:59:50.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.071+0000 7f14377fe700 1 -- 172.21.15.173:0/2873812179 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1428012330 con 0x7f14400f5410 2024-02-19T21:59:50.071 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:50.071 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:50.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.072+0000 7f1445965700 1 -- 172.21.15.173:0/2873812179 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f142c02ff10 msgr2=0x7f142c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:50.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.072+0000 7f1445965700 1 --2- 172.21.15.173:0/2873812179 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f142c02ff10 0x7f142c0323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f14300096a0 tx=0x7f1430006b40).stop 2024-02-19T21:59:50.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.073+0000 7f1445965700 1 -- 172.21.15.173:0/2873812179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14400f5410 msgr2=0x7f14400fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:50.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.073+0000 7f1445965700 1 --2- 172.21.15.173:0/2873812179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14400f5410 0x7f14400fd4e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f1428005010 tx=0x7f142800b040).stop 2024-02-19T21:59:50.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.073+0000 7f1445965700 1 -- 172.21.15.173:0/2873812179 shutdown_connections 2024-02-19T21:59:50.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.073+0000 7f1445965700 1 --2- 172.21.15.173:0/2873812179 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f142c02ff10 0x7f142c0323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:50.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.073+0000 7f1445965700 1 --2- 172.21.15.173:0/2873812179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14400f5410 0x7f14400fd4e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:50.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.073+0000 7f1445965700 1 -- 172.21.15.173:0/2873812179 >> 172.21.15.173:0/2873812179 conn(0x7f14400f0b70 msgr2=0x7f144005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:50.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.073+0000 7f1445965700 1 -- 172.21.15.173:0/2873812179 shutdown_connections 2024-02-19T21:59:50.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:50.073+0000 7f1445965700 1 -- 172.21.15.173:0/2873812179 wait complete. 2024-02-19T21:59:50.073 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:50.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:48.487477+0000 mgr.smithi143.bissbc (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:50.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:50.072270+0000 mon.smithi143 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.173:0/2873812179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:51.780 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:51.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:52.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:50.487676+0000 mgr.smithi143.bissbc (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:53.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.787+0000 7f3766bee700 1 -- 172.21.15.173:0/185763419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37600601a0 msgr2=0x7f37600f8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:53.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.787+0000 7f3766bee700 1 --2- 172.21.15.173:0/185763419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37600601a0 0x7f37600f8f60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f375c009fb0 tx=0x7f375c009b30).stop 2024-02-19T21:59:53.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.787+0000 7f3766bee700 1 -- 172.21.15.173:0/185763419 shutdown_connections 2024-02-19T21:59:53.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.787+0000 7f3766bee700 1 --2- 172.21.15.173:0/185763419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37600601a0 0x7f37600f8f60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:53.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.787+0000 7f3766bee700 1 -- 172.21.15.173:0/185763419 >> 172.21.15.173:0/185763419 conn(0x7f37600f09a0 msgr2=0x7f37600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:53.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.787+0000 7f3766bee700 1 -- 172.21.15.173:0/185763419 shutdown_connections 2024-02-19T21:59:53.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.787+0000 7f3766bee700 1 -- 172.21.15.173:0/185763419 wait complete. 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.787+0000 7f3766bee700 1 Processor -- start 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.787+0000 7f3766bee700 1 -- start start 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f3766bee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37600601a0 0x7f37600f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f3766bee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37600f4e60 con 0x7f37600601a0 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f3765bec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37600601a0 0x7f37600f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f3765bec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37600601a0 0x7f37600f69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:42924/0 (socket says 172.21.15.173:42924) 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f3765bec700 1 -- 172.21.15.173:0/3241610968 learned_addr learned my addr 172.21.15.173:0/3241610968 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f3765bec700 1 -- 172.21.15.173:0/3241610968 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c005040 con 0x7f37600601a0 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f3765bec700 1 --2- 172.21.15.173:0/3241610968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37600601a0 0x7f37600f69e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f375c005010 tx=0x7f375c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f37567fc700 1 -- 172.21.15.173:0/3241610968 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f375c009320 con 0x7f37600601a0 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f37567fc700 1 -- 172.21.15.173:0/3241610968 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f375c013380 con 0x7f37600601a0 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f3766bee700 1 -- 172.21.15.173:0/3241610968 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37600f5000 con 0x7f37600601a0 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f37567fc700 1 -- 172.21.15.173:0/3241610968 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f375c014e50 con 0x7f37600601a0 2024-02-19T21:59:53.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.788+0000 7f3766bee700 1 -- 172.21.15.173:0/3241610968 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37600f5490 con 0x7f37600601a0 2024-02-19T21:59:53.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.789+0000 7f37567fc700 1 -- 172.21.15.173:0/3241610968 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f375c0263a0 con 0x7f37600601a0 2024-02-19T21:59:53.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.789+0000 7f37567fc700 1 --2- 172.21.15.173:0/3241610968 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f374c02ff10 0x7f374c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:53.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.789+0000 7f37567fc700 1 -- 172.21.15.173:0/3241610968 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f375c040fc0 con 0x7f37600601a0 2024-02-19T21:59:53.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.789+0000 7f3766bee700 1 -- 172.21.15.173:0/3241610968 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3760059100 con 0x7f37600601a0 2024-02-19T21:59:53.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.789+0000 7f37653eb700 1 --2- 172.21.15.173:0/3241610968 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f374c02ff10 0x7f374c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:53.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.790+0000 7f37653eb700 1 --2- 172.21.15.173:0/3241610968 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f374c02ff10 0x7f374c0323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f37500096a0 tx=0x7f3750006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:53.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.795+0000 7f37567fc700 1 -- 172.21.15.173:0/3241610968 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f375c012920 con 0x7f37600601a0 2024-02-19T21:59:53.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:53.999+0000 7f3766bee700 1 -- 172.21.15.173:0/3241610968 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37600fbe10 con 0x7f37600601a0 2024-02-19T21:59:53.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:54.000+0000 7f37567fc700 1 -- 172.21.15.173:0/3241610968 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f375c012330 con 0x7f37600601a0 2024-02-19T21:59:54.000 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:54.000 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:54.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:54.001+0000 7f3766bee700 1 -- 172.21.15.173:0/3241610968 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f374c02ff10 msgr2=0x7f374c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:54.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:54.001+0000 7f3766bee700 1 --2- 172.21.15.173:0/3241610968 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f374c02ff10 0x7f374c0323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f37500096a0 tx=0x7f3750006b40).stop 2024-02-19T21:59:54.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:54.001+0000 7f3766bee700 1 -- 172.21.15.173:0/3241610968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37600601a0 msgr2=0x7f37600f69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:54.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:54.001+0000 7f3766bee700 1 --2- 172.21.15.173:0/3241610968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37600601a0 0x7f37600f69e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f375c005010 tx=0x7f375c00b040).stop 2024-02-19T21:59:54.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:54.001+0000 7f3766bee700 1 -- 172.21.15.173:0/3241610968 shutdown_connections 2024-02-19T21:59:54.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:54.001+0000 7f3766bee700 1 --2- 172.21.15.173:0/3241610968 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f374c02ff10 0x7f374c0323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:54.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:54.001+0000 7f3766bee700 1 --2- 172.21.15.173:0/3241610968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37600601a0 0x7f37600f69e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:54.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:54.001+0000 7f3766bee700 1 -- 172.21.15.173:0/3241610968 >> 172.21.15.173:0/3241610968 conn(0x7f37600f09a0 msgr2=0x7f37600f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:54.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:54.001+0000 7f3766bee700 1 -- 172.21.15.173:0/3241610968 shutdown_connections 2024-02-19T21:59:54.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:54.002+0000 7f3766bee700 1 -- 172.21.15.173:0/3241610968 wait complete. 2024-02-19T21:59:54.002 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:54.542 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:52.487926+0000 mgr.smithi143.bissbc (mgr.14162) 171 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:54.542 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:54.001038+0000 mon.smithi143 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.173:0/3241610968' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:54.543 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:54.162568+0000 mon.smithi143 (mon.0) 213 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T21:59:55.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:55.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:55.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:55 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T21:59:54.163859+0000 mgr.smithi143.bissbc (mgr.14162) 172 : cephadm [INF] Deploying daemon node-exporter.smithi143 on smithi143 2024-02-19T21:59:55.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:55 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:54.488148+0000 mgr.smithi143.bissbc (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:56.488653+0000 mgr.smithi143.bissbc (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:57.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.950+0000 7f7209dd2700 1 -- 172.21.15.173:0/866629667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72040f5410 msgr2=0x7f72040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:57.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.950+0000 7f7209dd2700 1 --2- 172.21.15.173:0/866629667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72040f5410 0x7f72040f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f71f4009fb0 tx=0x7f71f4009b30).stop 2024-02-19T21:59:57.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.950+0000 7f7209dd2700 1 -- 172.21.15.173:0/866629667 shutdown_connections 2024-02-19T21:59:57.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.950+0000 7f7209dd2700 1 --2- 172.21.15.173:0/866629667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72040f5410 0x7f72040f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:57.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.950+0000 7f7209dd2700 1 -- 172.21.15.173:0/866629667 >> 172.21.15.173:0/866629667 conn(0x7f72040f0b70 msgr2=0x7f72040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:57.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.950+0000 7f7209dd2700 1 -- 172.21.15.173:0/866629667 shutdown_connections 2024-02-19T21:59:57.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.950+0000 7f7209dd2700 1 -- 172.21.15.173:0/866629667 wait complete. 2024-02-19T21:59:57.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.950+0000 7f7209dd2700 1 Processor -- start 2024-02-19T21:59:57.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.950+0000 7f7209dd2700 1 -- start start 2024-02-19T21:59:57.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.951+0000 7f7209dd2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72040f5410 0x7f72040601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:57.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.951+0000 7f7209dd2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7204109500 con 0x7f72040f5410 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.951+0000 7f72037fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72040f5410 0x7f72040601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.951+0000 7f72037fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72040f5410 0x7f72040601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:42938/0 (socket says 172.21.15.173:42938) 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.951+0000 7f72037fe700 1 -- 172.21.15.173:0/1053326540 learned_addr learned my addr 172.21.15.173:0/1053326540 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.951+0000 7f72037fe700 1 -- 172.21.15.173:0/1053326540 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f4005040 con 0x7f72040f5410 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.951+0000 7f72037fe700 1 --2- 172.21.15.173:0/1053326540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72040f5410 0x7f72040601b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f71f4005010 tx=0x7f71f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.951+0000 7f71ebfff700 1 -- 172.21.15.173:0/1053326540 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71f4009320 con 0x7f72040f5410 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.952+0000 7f7209dd2700 1 -- 172.21.15.173:0/1053326540 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7204109640 con 0x7f72040f5410 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.952+0000 7f71ebfff700 1 -- 172.21.15.173:0/1053326540 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71f4013380 con 0x7f72040f5410 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.952+0000 7f71ebfff700 1 -- 172.21.15.173:0/1053326540 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71f4014e50 con 0x7f72040f5410 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.952+0000 7f7209dd2700 1 -- 172.21.15.173:0/1053326540 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7204109b30 con 0x7f72040f5410 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.952+0000 7f71ebfff700 1 -- 172.21.15.173:0/1053326540 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71f4014420 con 0x7f72040f5410 2024-02-19T21:59:57.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.952+0000 7f7209dd2700 1 -- 172.21.15.173:0/1053326540 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7204045ad0 con 0x7f72040f5410 2024-02-19T21:59:57.954 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.952+0000 7f71ebfff700 1 --2- 172.21.15.173:0/1053326540 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f71ec02ff10 0x7f71ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:57.954 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.953+0000 7f71ebfff700 1 -- 172.21.15.173:0/1053326540 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71f4040e30 con 0x7f72040f5410 2024-02-19T21:59:57.954 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.953+0000 7f7202ffd700 1 --2- 172.21.15.173:0/1053326540 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f71ec02ff10 0x7f71ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:57.954 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.953+0000 7f7202ffd700 1 --2- 172.21.15.173:0/1053326540 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f71ec02ff10 0x7f71ec0323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f71f80096a0 tx=0x7f71f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:57.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:57.958+0000 7f71ebfff700 1 -- 172.21.15.173:0/1053326540 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71f40095f0 con 0x7f72040f5410 2024-02-19T21:59:58.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.160+0000 7f7209dd2700 1 -- 172.21.15.173:0/1053326540 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f720410a360 con 0x7f72040f5410 2024-02-19T21:59:58.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.161+0000 7f71ebfff700 1 -- 172.21.15.173:0/1053326540 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f71f401db70 con 0x7f72040f5410 2024-02-19T21:59:58.161 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T21:59:58.161 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:58.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.163+0000 7f7209dd2700 1 -- 172.21.15.173:0/1053326540 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f71ec02ff10 msgr2=0x7f71ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:58.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.163+0000 7f7209dd2700 1 --2- 172.21.15.173:0/1053326540 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f71ec02ff10 0x7f71ec0323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f71f80096a0 tx=0x7f71f8006b40).stop 2024-02-19T21:59:58.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.163+0000 7f7209dd2700 1 -- 172.21.15.173:0/1053326540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72040f5410 msgr2=0x7f72040601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:58.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.163+0000 7f7209dd2700 1 --2- 172.21.15.173:0/1053326540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72040f5410 0x7f72040601b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f71f4005010 tx=0x7f71f400b040).stop 2024-02-19T21:59:58.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.163+0000 7f7209dd2700 1 -- 172.21.15.173:0/1053326540 shutdown_connections 2024-02-19T21:59:58.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.163+0000 7f7209dd2700 1 --2- 172.21.15.173:0/1053326540 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f71ec02ff10 0x7f71ec0323d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:58.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.163+0000 7f7209dd2700 1 --2- 172.21.15.173:0/1053326540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72040f5410 0x7f72040601b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:58.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.163+0000 7f7209dd2700 1 -- 172.21.15.173:0/1053326540 >> 172.21.15.173:0/1053326540 conn(0x7f72040f0b70 msgr2=0x7f72040f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:58.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.163+0000 7f7209dd2700 1 -- 172.21.15.173:0/1053326540 shutdown_connections 2024-02-19T21:59:58.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T21:59:58.163+0000 7f7209dd2700 1 -- 172.21.15.173:0/1053326540 wait complete. 2024-02-19T21:59:58.164 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T21:59:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 21:59:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T21:59:58.162365+0000 mon.smithi143 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.173:0/1053326540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:59.810 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:59.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T21:59:58.489087+0000 mgr.smithi143.bissbc (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:01.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.889+0000 7f083bf3f700 1 -- 172.21.15.173:0/2228435140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08340f6510 msgr2=0x7f08340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:01.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.889+0000 7f083bf3f700 1 --2- 172.21.15.173:0/2228435140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08340f6510 0x7f08340f6930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f0824009fb0 tx=0x7f0824009b30).stop 2024-02-19T22:00:01.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.889+0000 7f083bf3f700 1 -- 172.21.15.173:0/2228435140 shutdown_connections 2024-02-19T22:00:01.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.889+0000 7f083bf3f700 1 --2- 172.21.15.173:0/2228435140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08340f6510 0x7f08340f6930 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:01.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.889+0000 7f083bf3f700 1 -- 172.21.15.173:0/2228435140 >> 172.21.15.173:0/2228435140 conn(0x7f08340f1c70 msgr2=0x7f08340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:01.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.889+0000 7f083bf3f700 1 -- 172.21.15.173:0/2228435140 shutdown_connections 2024-02-19T22:00:01.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.889+0000 7f083bf3f700 1 -- 172.21.15.173:0/2228435140 wait complete. 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.890+0000 7f083bf3f700 1 Processor -- start 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.890+0000 7f083bf3f700 1 -- start start 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.890+0000 7f083bf3f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08340f6510 0x7f08341067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.890+0000 7f083bf3f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0834106cd0 con 0x7f08340f6510 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.890+0000 7f0839cdb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08340f6510 0x7f08341067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.890+0000 7f0839cdb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08340f6510 0x7f08341067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:56972/0 (socket says 172.21.15.173:56972) 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.890+0000 7f0839cdb700 1 -- 172.21.15.173:0/943224925 learned_addr learned my addr 172.21.15.173:0/943224925 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.891+0000 7f0839cdb700 1 -- 172.21.15.173:0/943224925 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0824005040 con 0x7f08340f6510 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.891+0000 7f0839cdb700 1 --2- 172.21.15.173:0/943224925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08340f6510 0x7f08341067e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0824005010 tx=0x7f082400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.891+0000 7f082a7fc700 1 -- 172.21.15.173:0/943224925 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0824009320 con 0x7f08340f6510 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.891+0000 7f082a7fc700 1 -- 172.21.15.173:0/943224925 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0824013380 con 0x7f08340f6510 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.891+0000 7f083bf3f700 1 -- 172.21.15.173:0/943224925 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0834106ed0 con 0x7f08340f6510 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.891+0000 7f082a7fc700 1 -- 172.21.15.173:0/943224925 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0824014e50 con 0x7f08340f6510 2024-02-19T22:00:01.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.891+0000 7f083bf3f700 1 -- 172.21.15.173:0/943224925 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0834107360 con 0x7f08340f6510 2024-02-19T22:00:01.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.892+0000 7f082a7fc700 1 -- 172.21.15.173:0/943224925 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08240263a0 con 0x7f08340f6510 2024-02-19T22:00:01.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.892+0000 7f082a7fc700 1 --2- 172.21.15.173:0/943224925 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f082002ff10 0x7f08200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:01.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.892+0000 7f082a7fc700 1 -- 172.21.15.173:0/943224925 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0824041250 con 0x7f08340f6510 2024-02-19T22:00:01.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.892+0000 7f083bf3f700 1 -- 172.21.15.173:0/943224925 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0834045ad0 con 0x7f08340f6510 2024-02-19T22:00:01.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.892+0000 7f08394da700 1 --2- 172.21.15.173:0/943224925 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f082002ff10 0x7f08200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:01.895 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.893+0000 7f08394da700 1 --2- 172.21.15.173:0/943224925 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f082002ff10 0x7f08200323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f08300096a0 tx=0x7f0830006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:01.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:01.897+0000 7f082a7fc700 1 -- 172.21.15.173:0/943224925 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0824012920 con 0x7f08340f6510 2024-02-19T22:00:02.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.103+0000 7f083bf3f700 1 -- 172.21.15.173:0/943224925 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0834108210 con 0x7f08340f6510 2024-02-19T22:00:02.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.103+0000 7f082a7fc700 1 -- 172.21.15.173:0/943224925 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0824012330 con 0x7f08340f6510 2024-02-19T22:00:02.104 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:02.105 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:02.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.105+0000 7f083bf3f700 1 -- 172.21.15.173:0/943224925 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f082002ff10 msgr2=0x7f08200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:02.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.105+0000 7f083bf3f700 1 --2- 172.21.15.173:0/943224925 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f082002ff10 0x7f08200323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f08300096a0 tx=0x7f0830006b40).stop 2024-02-19T22:00:02.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.105+0000 7f083bf3f700 1 -- 172.21.15.173:0/943224925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08340f6510 msgr2=0x7f08341067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:02.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.105+0000 7f083bf3f700 1 --2- 172.21.15.173:0/943224925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08340f6510 0x7f08341067e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0824005010 tx=0x7f082400b040).stop 2024-02-19T22:00:02.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.105+0000 7f083bf3f700 1 -- 172.21.15.173:0/943224925 shutdown_connections 2024-02-19T22:00:02.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.105+0000 7f083bf3f700 1 --2- 172.21.15.173:0/943224925 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f082002ff10 0x7f08200323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:02.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.105+0000 7f083bf3f700 1 --2- 172.21.15.173:0/943224925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08340f6510 0x7f08341067e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:02.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.105+0000 7f083bf3f700 1 -- 172.21.15.173:0/943224925 >> 172.21.15.173:0/943224925 conn(0x7f08340f1c70 msgr2=0x7f08340f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:02.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.105+0000 7f083bf3f700 1 -- 172.21.15.173:0/943224925 shutdown_connections 2024-02-19T22:00:02.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:02.105+0000 7f083bf3f700 1 -- 172.21.15.173:0/943224925 wait complete. 2024-02-19T22:00:02.107 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:00.489381+0000 mgr.smithi143.bissbc (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:02.105808+0000 mon.smithi143 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.173:0/943224925' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:03.892 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:03.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:02.489766+0000 mgr.smithi143.bissbc (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:05.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.819+0000 7f3afb59e700 1 -- 172.21.15.173:0/1162880965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc060180 msgr2=0x7f3afc0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:05.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.819+0000 7f3afb59e700 1 --2- 172.21.15.173:0/1162880965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc060180 0x7f3afc0f8f80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f3af0009fb0 tx=0x7f3af0009b30).stop 2024-02-19T22:00:05.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afb59e700 1 -- 172.21.15.173:0/1162880965 shutdown_connections 2024-02-19T22:00:05.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afb59e700 1 --2- 172.21.15.173:0/1162880965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc060180 0x7f3afc0f8f80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:05.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afb59e700 1 -- 172.21.15.173:0/1162880965 >> 172.21.15.173:0/1162880965 conn(0x7f3afc0f09a0 msgr2=0x7f3afc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:05.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afb59e700 1 -- 172.21.15.173:0/1162880965 shutdown_connections 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afb59e700 1 -- 172.21.15.173:0/1162880965 wait complete. 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afb59e700 1 Processor -- start 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afb59e700 1 -- start start 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afb59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc060180 0x7f3afc0f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afb59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc0f4e80 con 0x7f3afc060180 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afa59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc060180 0x7f3afc0f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afa59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc060180 0x7f3afc0f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:56986/0 (socket says 172.21.15.173:56986) 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.820+0000 7f3afa59c700 1 -- 172.21.15.173:0/1510767903 learned_addr learned my addr 172.21.15.173:0/1510767903 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.821+0000 7f3afa59c700 1 -- 172.21.15.173:0/1510767903 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af0005040 con 0x7f3afc060180 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.821+0000 7f3afa59c700 1 --2- 172.21.15.173:0/1510767903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc060180 0x7f3afc0f6a00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f3af0005010 tx=0x7f3af000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.821+0000 7f3aeaffd700 1 -- 172.21.15.173:0/1510767903 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3af0009320 con 0x7f3afc060180 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.821+0000 7f3afb59e700 1 -- 172.21.15.173:0/1510767903 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3afc0f5020 con 0x7f3afc060180 2024-02-19T22:00:05.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.821+0000 7f3aeaffd700 1 -- 172.21.15.173:0/1510767903 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3af0013380 con 0x7f3afc060180 2024-02-19T22:00:05.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.821+0000 7f3aeaffd700 1 -- 172.21.15.173:0/1510767903 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3af0014e50 con 0x7f3afc060180 2024-02-19T22:00:05.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.821+0000 7f3afb59e700 1 -- 172.21.15.173:0/1510767903 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3afc0f54b0 con 0x7f3afc060180 2024-02-19T22:00:05.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.822+0000 7f3afb59e700 1 -- 172.21.15.173:0/1510767903 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3afc059100 con 0x7f3afc060180 2024-02-19T22:00:05.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.822+0000 7f3aeaffd700 1 -- 172.21.15.173:0/1510767903 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3af0014420 con 0x7f3afc060180 2024-02-19T22:00:05.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.822+0000 7f3aeaffd700 1 --2- 172.21.15.173:0/1510767903 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ae402ff10 0x7f3ae40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:05.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.822+0000 7f3aeaffd700 1 -- 172.21.15.173:0/1510767903 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3af0040eb0 con 0x7f3afc060180 2024-02-19T22:00:05.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.822+0000 7f3af9d9b700 1 --2- 172.21.15.173:0/1510767903 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ae402ff10 0x7f3ae40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:05.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.823+0000 7f3af9d9b700 1 --2- 172.21.15.173:0/1510767903 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ae402ff10 0x7f3ae40323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f3aec0096a0 tx=0x7f3aec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:05.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:05.827+0000 7f3aeaffd700 1 -- 172.21.15.173:0/1510767903 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3af00095f0 con 0x7f3afc060180 2024-02-19T22:00:06.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.032+0000 7f3afb59e700 1 -- 172.21.15.173:0/1510767903 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3afc0fbe20 con 0x7f3afc060180 2024-02-19T22:00:06.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.033+0000 7f3aeaffd700 1 -- 172.21.15.173:0/1510767903 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3af001db70 con 0x7f3afc060180 2024-02-19T22:00:06.032 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:06.032 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:06.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.034+0000 7f3afb59e700 1 -- 172.21.15.173:0/1510767903 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ae402ff10 msgr2=0x7f3ae40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:06.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.034+0000 7f3afb59e700 1 --2- 172.21.15.173:0/1510767903 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ae402ff10 0x7f3ae40323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f3aec0096a0 tx=0x7f3aec006b40).stop 2024-02-19T22:00:06.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.034+0000 7f3afb59e700 1 -- 172.21.15.173:0/1510767903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc060180 msgr2=0x7f3afc0f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:06.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.034+0000 7f3afb59e700 1 --2- 172.21.15.173:0/1510767903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc060180 0x7f3afc0f6a00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f3af0005010 tx=0x7f3af000b040).stop 2024-02-19T22:00:06.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.034+0000 7f3afb59e700 1 -- 172.21.15.173:0/1510767903 shutdown_connections 2024-02-19T22:00:06.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.034+0000 7f3afb59e700 1 --2- 172.21.15.173:0/1510767903 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ae402ff10 0x7f3ae40323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:06.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.034+0000 7f3afb59e700 1 --2- 172.21.15.173:0/1510767903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc060180 0x7f3afc0f6a00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:06.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.034+0000 7f3afb59e700 1 -- 172.21.15.173:0/1510767903 >> 172.21.15.173:0/1510767903 conn(0x7f3afc0f09a0 msgr2=0x7f3afc0f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:06.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.034+0000 7f3afb59e700 1 -- 172.21.15.173:0/1510767903 shutdown_connections 2024-02-19T22:00:06.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:06.034+0000 7f3afb59e700 1 -- 172.21.15.173:0/1510767903 wait complete. 2024-02-19T22:00:06.035 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:04.490105+0000 mgr.smithi143.bissbc (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:06.033645+0000 mon.smithi143 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.173:0/1510767903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:07.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:07.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:06.490486+0000 mgr.smithi143.bissbc (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:09.619 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.603+0000 7f952189a700 1 -- 172.21.15.173:0/3787894895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c0f4b80 msgr2=0x7f951c0f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:09.619 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.603+0000 7f952189a700 1 --2- 172.21.15.173:0/3787894895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c0f4b80 0x7f951c0f4fa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f950c009fb0 tx=0x7f950c009b30).stop 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.603+0000 7f952189a700 1 -- 172.21.15.173:0/3787894895 shutdown_connections 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.603+0000 7f952189a700 1 --2- 172.21.15.173:0/3787894895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c0f4b80 0x7f951c0f4fa0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.603+0000 7f952189a700 1 -- 172.21.15.173:0/3787894895 >> 172.21.15.173:0/3787894895 conn(0x7f951c0f02e0 msgr2=0x7f951c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.603+0000 7f952189a700 1 -- 172.21.15.173:0/3787894895 shutdown_connections 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.603+0000 7f952189a700 1 -- 172.21.15.173:0/3787894895 wait complete. 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.603+0000 7f952189a700 1 Processor -- start 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.603+0000 7f952189a700 1 -- start start 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.604+0000 7f952189a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c0f4b80 0x7f951c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.604+0000 7f952189a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c0f1850 con 0x7f951c0f4b80 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.604+0000 7f951affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c0f4b80 0x7f951c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.604+0000 7f951affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c0f4b80 0x7f951c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:57012/0 (socket says 172.21.15.173:57012) 2024-02-19T22:00:09.620 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.604+0000 7f951affd700 1 -- 172.21.15.173:0/618817399 learned_addr learned my addr 172.21.15.173:0/618817399 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.604+0000 7f951affd700 1 -- 172.21.15.173:0/618817399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c005040 con 0x7f951c0f4b80 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.604+0000 7f951affd700 1 --2- 172.21.15.173:0/618817399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c0f4b80 0x7f951c1094e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f950c005010 tx=0x7f950c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.604+0000 7f95037fe700 1 -- 172.21.15.173:0/618817399 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f950c009320 con 0x7f951c0f4b80 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.605+0000 7f952189a700 1 -- 172.21.15.173:0/618817399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951c0f1a50 con 0x7f951c0f4b80 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.605+0000 7f95037fe700 1 -- 172.21.15.173:0/618817399 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f950c013380 con 0x7f951c0f4b80 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.605+0000 7f95037fe700 1 -- 172.21.15.173:0/618817399 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f950c014e50 con 0x7f951c0f4b80 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.605+0000 7f952189a700 1 -- 172.21.15.173:0/618817399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951c109b90 con 0x7f951c0f4b80 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.605+0000 7f95037fe700 1 -- 172.21.15.173:0/618817399 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f950c0095f0 con 0x7f951c0f4b80 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.605+0000 7f952189a700 1 -- 172.21.15.173:0/618817399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951c059100 con 0x7f951c0f4b80 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.605+0000 7f95037fe700 1 --2- 172.21.15.173:0/618817399 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f950402ff10 0x7f95040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.606+0000 7f95037fe700 1 -- 172.21.15.173:0/618817399 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f950c040f80 con 0x7f951c0f4b80 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.606+0000 7f951a7fc700 1 --2- 172.21.15.173:0/618817399 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f950402ff10 0x7f95040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.606+0000 7f951a7fc700 1 --2- 172.21.15.173:0/618817399 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f950402ff10 0x7f95040323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f95100096a0 tx=0x7f9510006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:09.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.610+0000 7f95037fe700 1 -- 172.21.15.173:0/618817399 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f950c014420 con 0x7f951c0f4b80 2024-02-19T22:00:09.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.814+0000 7f952189a700 1 -- 172.21.15.173:0/618817399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f951c027a00 con 0x7f951c0f4b80 2024-02-19T22:00:09.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.815+0000 7f95037fe700 1 -- 172.21.15.173:0/618817399 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f950c010230 con 0x7f951c0f4b80 2024-02-19T22:00:09.815 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:09.815 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:09.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.817+0000 7f952189a700 1 -- 172.21.15.173:0/618817399 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f950402ff10 msgr2=0x7f95040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:09.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.817+0000 7f952189a700 1 --2- 172.21.15.173:0/618817399 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f950402ff10 0x7f95040323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f95100096a0 tx=0x7f9510006b40).stop 2024-02-19T22:00:09.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.817+0000 7f952189a700 1 -- 172.21.15.173:0/618817399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c0f4b80 msgr2=0x7f951c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:09.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.817+0000 7f952189a700 1 --2- 172.21.15.173:0/618817399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c0f4b80 0x7f951c1094e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f950c005010 tx=0x7f950c00b040).stop 2024-02-19T22:00:09.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.817+0000 7f952189a700 1 -- 172.21.15.173:0/618817399 shutdown_connections 2024-02-19T22:00:09.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.817+0000 7f952189a700 1 --2- 172.21.15.173:0/618817399 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f950402ff10 0x7f95040323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:09.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.817+0000 7f952189a700 1 --2- 172.21.15.173:0/618817399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c0f4b80 0x7f951c1094e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:09.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.817+0000 7f952189a700 1 -- 172.21.15.173:0/618817399 >> 172.21.15.173:0/618817399 conn(0x7f951c0f02e0 msgr2=0x7f951c0f24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:09.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.817+0000 7f952189a700 1 -- 172.21.15.173:0/618817399 shutdown_connections 2024-02-19T22:00:09.817 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:09.817+0000 7f952189a700 1 -- 172.21.15.173:0/618817399 wait complete. 2024-02-19T22:00:09.817 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:08.490809+0000 mgr.smithi143.bissbc (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:09.816038+0000 mon.smithi143 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.173:0/618817399' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:11.596 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:11.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:12.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:10.491098+0000 mgr.smithi143.bissbc (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:13.591 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.590+0000 7fb0ed989700 1 -- 172.21.15.173:0/2765292021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e80f51a0 msgr2=0x7fb0e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.590+0000 7fb0ed989700 1 --2- 172.21.15.173:0/2765292021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e80f51a0 0x7fb0e80f55c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fb0dc009fb0 tx=0x7fb0dc009b30).stop 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.590+0000 7fb0ed989700 1 -- 172.21.15.173:0/2765292021 shutdown_connections 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.590+0000 7fb0ed989700 1 --2- 172.21.15.173:0/2765292021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e80f51a0 0x7fb0e80f55c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.590+0000 7fb0ed989700 1 -- 172.21.15.173:0/2765292021 >> 172.21.15.173:0/2765292021 conn(0x7fb0e80f09a0 msgr2=0x7fb0e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.591+0000 7fb0ed989700 1 -- 172.21.15.173:0/2765292021 shutdown_connections 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.591+0000 7fb0ed989700 1 -- 172.21.15.173:0/2765292021 wait complete. 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.591+0000 7fb0ed989700 1 Processor -- start 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.591+0000 7fb0ed989700 1 -- start start 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.591+0000 7fb0ed989700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e80f51a0 0x7fb0e81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.591+0000 7fb0ed989700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8106ac0 con 0x7fb0e80f51a0 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.592+0000 7fb0ec987700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e80f51a0 0x7fb0e81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.592+0000 7fb0ec987700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e80f51a0 0x7fb0e81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41370/0 (socket says 172.21.15.173:41370) 2024-02-19T22:00:13.592 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.592+0000 7fb0ec987700 1 -- 172.21.15.173:0/1730108659 learned_addr learned my addr 172.21.15.173:0/1730108659 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:13.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.592+0000 7fb0ec987700 1 -- 172.21.15.173:0/1730108659 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc005040 con 0x7fb0e80f51a0 2024-02-19T22:00:13.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.592+0000 7fb0ec987700 1 --2- 172.21.15.173:0/1730108659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e80f51a0 0x7fb0e81065d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fb0dc009830 tx=0x7fb0dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:13.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.592+0000 7fb0e57fa700 1 -- 172.21.15.173:0/1730108659 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0dc009320 con 0x7fb0e80f51a0 2024-02-19T22:00:13.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.592+0000 7fb0ed989700 1 -- 172.21.15.173:0/1730108659 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e8106cc0 con 0x7fb0e80f51a0 2024-02-19T22:00:13.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.592+0000 7fb0e57fa700 1 -- 172.21.15.173:0/1730108659 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0dc013380 con 0x7fb0e80f51a0 2024-02-19T22:00:13.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.592+0000 7fb0ed989700 1 -- 172.21.15.173:0/1730108659 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e8107150 con 0x7fb0e80f51a0 2024-02-19T22:00:13.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.592+0000 7fb0e57fa700 1 -- 172.21.15.173:0/1730108659 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0dc01eea0 con 0x7fb0e80f51a0 2024-02-19T22:00:13.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.593+0000 7fb0e57fa700 1 -- 172.21.15.173:0/1730108659 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0dc014420 con 0x7fb0e80f51a0 2024-02-19T22:00:13.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.593+0000 7fb0e57fa700 1 --2- 172.21.15.173:0/1730108659 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb0d802ff60 0x7fb0d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:13.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.593+0000 7fb0e57fa700 1 -- 172.21.15.173:0/1730108659 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0dc040e70 con 0x7fb0e80f51a0 2024-02-19T22:00:13.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.593+0000 7fb0ed989700 1 -- 172.21.15.173:0/1730108659 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e8045ad0 con 0x7fb0e80f51a0 2024-02-19T22:00:13.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.593+0000 7fb0e7fff700 1 --2- 172.21.15.173:0/1730108659 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb0d802ff60 0x7fb0d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:13.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.594+0000 7fb0e7fff700 1 --2- 172.21.15.173:0/1730108659 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb0d802ff60 0x7fb0d8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb0d40096a0 tx=0x7fb0d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:13.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.598+0000 7fb0e57fa700 1 -- 172.21.15.173:0/1730108659 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0dc0273f0 con 0x7fb0e80f51a0 2024-02-19T22:00:13.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.806+0000 7fb0ed989700 1 -- 172.21.15.173:0/1730108659 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0e81000e0 con 0x7fb0e80f51a0 2024-02-19T22:00:13.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.807+0000 7fb0e57fa700 1 -- 172.21.15.173:0/1730108659 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb0dc01a230 con 0x7fb0e80f51a0 2024-02-19T22:00:13.807 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:13.807 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:13.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.809+0000 7fb0ed989700 1 -- 172.21.15.173:0/1730108659 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb0d802ff60 msgr2=0x7fb0d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:13.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.809+0000 7fb0ed989700 1 --2- 172.21.15.173:0/1730108659 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb0d802ff60 0x7fb0d8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb0d40096a0 tx=0x7fb0d4006b40).stop 2024-02-19T22:00:13.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.809+0000 7fb0ed989700 1 -- 172.21.15.173:0/1730108659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e80f51a0 msgr2=0x7fb0e81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:13.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.809+0000 7fb0ed989700 1 --2- 172.21.15.173:0/1730108659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e80f51a0 0x7fb0e81065d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fb0dc009830 tx=0x7fb0dc00b040).stop 2024-02-19T22:00:13.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.809+0000 7fb0ed989700 1 -- 172.21.15.173:0/1730108659 shutdown_connections 2024-02-19T22:00:13.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.809+0000 7fb0ed989700 1 --2- 172.21.15.173:0/1730108659 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb0d802ff60 0x7fb0d8032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:13.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.809+0000 7fb0ed989700 1 --2- 172.21.15.173:0/1730108659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e80f51a0 0x7fb0e81065d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:13.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.809+0000 7fb0ed989700 1 -- 172.21.15.173:0/1730108659 >> 172.21.15.173:0/1730108659 conn(0x7fb0e80f09a0 msgr2=0x7fb0e805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:13.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.809+0000 7fb0ed989700 1 -- 172.21.15.173:0/1730108659 shutdown_connections 2024-02-19T22:00:13.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:13.809+0000 7fb0ed989700 1 -- 172.21.15.173:0/1730108659 wait complete. 2024-02-19T22:00:13.809 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:12.491504+0000 mgr.smithi143.bissbc (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:13.808182+0000 mon.smithi143 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.173:0/1730108659' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:15.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:15.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:14.491879+0000 mgr.smithi143.bissbc (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:17.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.358+0000 7faf48251700 1 -- 172.21.15.173:0/2792728526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf40060200 msgr2=0x7faf400f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:17.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.358+0000 7faf48251700 1 --2- 172.21.15.173:0/2792728526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf40060200 0x7faf400f9170 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7faf3c009fb0 tx=0x7faf3c009b30).stop 2024-02-19T22:00:17.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.358+0000 7faf48251700 1 -- 172.21.15.173:0/2792728526 shutdown_connections 2024-02-19T22:00:17.360 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.358+0000 7faf48251700 1 --2- 172.21.15.173:0/2792728526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf40060200 0x7faf400f9170 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.358+0000 7faf48251700 1 -- 172.21.15.173:0/2792728526 >> 172.21.15.173:0/2792728526 conn(0x7faf400f0b70 msgr2=0x7faf400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.358+0000 7faf48251700 1 -- 172.21.15.173:0/2792728526 shutdown_connections 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.358+0000 7faf48251700 1 -- 172.21.15.173:0/2792728526 wait complete. 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.359+0000 7faf48251700 1 Processor -- start 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.359+0000 7faf48251700 1 -- start start 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.359+0000 7faf48251700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf40060200 0x7faf40106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.359+0000 7faf48251700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf40106c70 con 0x7faf40060200 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.359+0000 7faf45fed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf40060200 0x7faf40106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.359+0000 7faf45fed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf40060200 0x7faf40106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41394/0 (socket says 172.21.15.173:41394) 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.359+0000 7faf45fed700 1 -- 172.21.15.173:0/4101283053 learned_addr learned my addr 172.21.15.173:0/4101283053 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.359+0000 7faf45fed700 1 -- 172.21.15.173:0/4101283053 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c005040 con 0x7faf40060200 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.359+0000 7faf45fed700 1 --2- 172.21.15.173:0/4101283053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf40060200 0x7faf40106780 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7faf3c0089f0 tx=0x7faf3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.359+0000 7faf367fc700 1 -- 172.21.15.173:0/4101283053 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf3c009320 con 0x7faf40060200 2024-02-19T22:00:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.360+0000 7faf367fc700 1 -- 172.21.15.173:0/4101283053 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf3c013380 con 0x7faf40060200 2024-02-19T22:00:17.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.360+0000 7faf48251700 1 -- 172.21.15.173:0/4101283053 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf40106e70 con 0x7faf40060200 2024-02-19T22:00:17.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.360+0000 7faf367fc700 1 -- 172.21.15.173:0/4101283053 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf3c014e20 con 0x7faf40060200 2024-02-19T22:00:17.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.360+0000 7faf48251700 1 -- 172.21.15.173:0/4101283053 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf40107300 con 0x7faf40060200 2024-02-19T22:00:17.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.360+0000 7faf367fc700 1 -- 172.21.15.173:0/4101283053 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf3c014420 con 0x7faf40060200 2024-02-19T22:00:17.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.360+0000 7faf48251700 1 -- 172.21.15.173:0/4101283053 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf40045ad0 con 0x7faf40060200 2024-02-19T22:00:17.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.361+0000 7faf367fc700 1 --2- 172.21.15.173:0/4101283053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf2c02ff10 0x7faf2c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:17.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.361+0000 7faf367fc700 1 -- 172.21.15.173:0/4101283053 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf3c041010 con 0x7faf40060200 2024-02-19T22:00:17.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.361+0000 7faf457ec700 1 --2- 172.21.15.173:0/4101283053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf2c02ff10 0x7faf2c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:17.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.361+0000 7faf457ec700 1 --2- 172.21.15.173:0/4101283053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf2c02ff10 0x7faf2c0323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7faf300096a0 tx=0x7faf30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:17.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.365+0000 7faf367fc700 1 -- 172.21.15.173:0/4101283053 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf3c06d2b0 con 0x7faf40060200 2024-02-19T22:00:17.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.569+0000 7faf48251700 1 -- 172.21.15.173:0/4101283053 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf401081b0 con 0x7faf40060200 2024-02-19T22:00:17.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.570+0000 7faf367fc700 1 -- 172.21.15.173:0/4101283053 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faf3c0095f0 con 0x7faf40060200 2024-02-19T22:00:17.570 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:17.570 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:17.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.571+0000 7faf48251700 1 -- 172.21.15.173:0/4101283053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf2c02ff10 msgr2=0x7faf2c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:17.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.571+0000 7faf48251700 1 --2- 172.21.15.173:0/4101283053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf2c02ff10 0x7faf2c0323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7faf300096a0 tx=0x7faf30006b40).stop 2024-02-19T22:00:17.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.571+0000 7faf48251700 1 -- 172.21.15.173:0/4101283053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf40060200 msgr2=0x7faf40106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:17.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.571+0000 7faf48251700 1 --2- 172.21.15.173:0/4101283053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf40060200 0x7faf40106780 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7faf3c0089f0 tx=0x7faf3c00b040).stop 2024-02-19T22:00:17.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.571+0000 7faf48251700 1 -- 172.21.15.173:0/4101283053 shutdown_connections 2024-02-19T22:00:17.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.571+0000 7faf48251700 1 --2- 172.21.15.173:0/4101283053 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faf2c02ff10 0x7faf2c0323d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:17.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.571+0000 7faf48251700 1 --2- 172.21.15.173:0/4101283053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf40060200 0x7faf40106780 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:17.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.571+0000 7faf48251700 1 -- 172.21.15.173:0/4101283053 >> 172.21.15.173:0/4101283053 conn(0x7faf400f0b70 msgr2=0x7faf400f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:17.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.571+0000 7faf48251700 1 -- 172.21.15.173:0/4101283053 shutdown_connections 2024-02-19T22:00:17.572 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:17.571+0000 7faf48251700 1 -- 172.21.15.173:0/4101283053 wait complete. 2024-02-19T22:00:17.572 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:16.492125+0000 mgr.smithi143.bissbc (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:17.571009+0000 mon.smithi143 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.173:0/4101283053' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:19.281 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:19.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:18.492399+0000 mgr.smithi143.bissbc (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:21.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.355+0000 7f433031a700 1 -- 172.21.15.173:0/4071355688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43280f5410 msgr2=0x7f43280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:21.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.355+0000 7f433031a700 1 --2- 172.21.15.173:0/4071355688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43280f5410 0x7f43280f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f4324009fb0 tx=0x7f4324009b30).stop 2024-02-19T22:00:21.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.355+0000 7f433031a700 1 -- 172.21.15.173:0/4071355688 shutdown_connections 2024-02-19T22:00:21.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.355+0000 7f433031a700 1 --2- 172.21.15.173:0/4071355688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43280f5410 0x7f43280f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:21.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.355+0000 7f433031a700 1 -- 172.21.15.173:0/4071355688 >> 172.21.15.173:0/4071355688 conn(0x7f43280f0b70 msgr2=0x7f43280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:21.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.355+0000 7f433031a700 1 -- 172.21.15.173:0/4071355688 shutdown_connections 2024-02-19T22:00:21.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.355+0000 7f433031a700 1 -- 172.21.15.173:0/4071355688 wait complete. 2024-02-19T22:00:21.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.355+0000 7f433031a700 1 Processor -- start 2024-02-19T22:00:21.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.355+0000 7f433031a700 1 -- start start 2024-02-19T22:00:21.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.355+0000 7f433031a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43280f5410 0x7f43281067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:21.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.355+0000 7f433031a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4328106cd0 con 0x7f43280f5410 2024-02-19T22:00:21.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.356+0000 7f432e0b6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43280f5410 0x7f43281067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:21.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.356+0000 7f432e0b6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43280f5410 0x7f43281067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:57486/0 (socket says 172.21.15.173:57486) 2024-02-19T22:00:21.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.356+0000 7f432e0b6700 1 -- 172.21.15.173:0/718456351 learned_addr learned my addr 172.21.15.173:0/718456351 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.356+0000 7f432e0b6700 1 -- 172.21.15.173:0/718456351 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4324005040 con 0x7f43280f5410 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.356+0000 7f432e0b6700 1 --2- 172.21.15.173:0/718456351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43280f5410 0x7f43281067e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f4324005010 tx=0x7f432400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.356+0000 7f431effd700 1 -- 172.21.15.173:0/718456351 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4324009320 con 0x7f43280f5410 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.356+0000 7f433031a700 1 -- 172.21.15.173:0/718456351 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4328106ed0 con 0x7f43280f5410 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.356+0000 7f431effd700 1 -- 172.21.15.173:0/718456351 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4324013380 con 0x7f43280f5410 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.356+0000 7f431effd700 1 -- 172.21.15.173:0/718456351 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4324014e50 con 0x7f43280f5410 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.356+0000 7f433031a700 1 -- 172.21.15.173:0/718456351 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4328107360 con 0x7f43280f5410 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.357+0000 7f433031a700 1 -- 172.21.15.173:0/718456351 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4328045ad0 con 0x7f43280f5410 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.357+0000 7f431effd700 1 -- 172.21.15.173:0/718456351 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43240095f0 con 0x7f43280f5410 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.357+0000 7f431effd700 1 --2- 172.21.15.173:0/718456351 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f431402ff10 0x7f43140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.357+0000 7f431effd700 1 -- 172.21.15.173:0/718456351 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4324040e90 con 0x7f43280f5410 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.358+0000 7f432d8b5700 1 --2- 172.21.15.173:0/718456351 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f431402ff10 0x7f43140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:21.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.358+0000 7f432d8b5700 1 --2- 172.21.15.173:0/718456351 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f431402ff10 0x7f43140323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f43180096a0 tx=0x7f4318006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:21.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.362+0000 7f431effd700 1 -- 172.21.15.173:0/718456351 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4324014420 con 0x7f43280f5410 2024-02-19T22:00:21.568 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.569+0000 7f433031a700 1 -- 172.21.15.173:0/718456351 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4328108210 con 0x7f43280f5410 2024-02-19T22:00:21.569 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.569+0000 7f431effd700 1 -- 172.21.15.173:0/718456351 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4324010230 con 0x7f43280f5410 2024-02-19T22:00:21.569 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:21.570 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:21.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.571+0000 7f433031a700 1 -- 172.21.15.173:0/718456351 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f431402ff10 msgr2=0x7f43140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:21.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.571+0000 7f433031a700 1 --2- 172.21.15.173:0/718456351 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f431402ff10 0x7f43140323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f43180096a0 tx=0x7f4318006b40).stop 2024-02-19T22:00:21.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.571+0000 7f433031a700 1 -- 172.21.15.173:0/718456351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43280f5410 msgr2=0x7f43281067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:21.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.571+0000 7f433031a700 1 --2- 172.21.15.173:0/718456351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43280f5410 0x7f43281067e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f4324005010 tx=0x7f432400b040).stop 2024-02-19T22:00:21.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.571+0000 7f433031a700 1 -- 172.21.15.173:0/718456351 shutdown_connections 2024-02-19T22:00:21.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.571+0000 7f433031a700 1 --2- 172.21.15.173:0/718456351 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f431402ff10 0x7f43140323d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:21.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.571+0000 7f433031a700 1 --2- 172.21.15.173:0/718456351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43280f5410 0x7f43281067e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:21.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.571+0000 7f433031a700 1 -- 172.21.15.173:0/718456351 >> 172.21.15.173:0/718456351 conn(0x7f43280f0b70 msgr2=0x7f43280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:21.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.571+0000 7f433031a700 1 -- 172.21.15.173:0/718456351 shutdown_connections 2024-02-19T22:00:21.571 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:21.571+0000 7f433031a700 1 -- 172.21.15.173:0/718456351 wait complete. 2024-02-19T22:00:21.572 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:20.567296+0000 mon.smithi143 (mon.0) 220 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T22:00:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:20.567488+0000 mon.smithi143 (mon.0) 221 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:00:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:20.492717+0000 mgr.smithi143.bissbc (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:21.570595+0000 mon.smithi143 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.173:0/718456351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:23.229 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:23.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:24.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:22.492933+0000 mgr.smithi143.bissbc (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:25.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.165+0000 7fa63c82f700 1 -- 172.21.15.173:0/3803728239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6380f51a0 msgr2=0x7fa6380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:25.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.165+0000 7fa63c82f700 1 --2- 172.21.15.173:0/3803728239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6380f51a0 0x7fa6380f55c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fa628009fb0 tx=0x7fa628009b30).stop 2024-02-19T22:00:25.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.166+0000 7fa63c82f700 1 -- 172.21.15.173:0/3803728239 shutdown_connections 2024-02-19T22:00:25.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.166+0000 7fa63c82f700 1 --2- 172.21.15.173:0/3803728239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6380f51a0 0x7fa6380f55c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:25.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.166+0000 7fa63c82f700 1 -- 172.21.15.173:0/3803728239 >> 172.21.15.173:0/3803728239 conn(0x7fa6380f09a0 msgr2=0x7fa6380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:25.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.166+0000 7fa63c82f700 1 -- 172.21.15.173:0/3803728239 shutdown_connections 2024-02-19T22:00:25.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.166+0000 7fa63c82f700 1 -- 172.21.15.173:0/3803728239 wait complete. 2024-02-19T22:00:25.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.166+0000 7fa63c82f700 1 Processor -- start 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.166+0000 7fa63c82f700 1 -- start start 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.166+0000 7fa63c82f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6380f51a0 0x7fa6381065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.166+0000 7fa63c82f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa638106ac0 con 0x7fa6380f51a0 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.167+0000 7fa636d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6380f51a0 0x7fa6381065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.167+0000 7fa636d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6380f51a0 0x7fa6381065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:57496/0 (socket says 172.21.15.173:57496) 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.167+0000 7fa636d9d700 1 -- 172.21.15.173:0/2754586082 learned_addr learned my addr 172.21.15.173:0/2754586082 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.167+0000 7fa636d9d700 1 -- 172.21.15.173:0/2754586082 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa628005040 con 0x7fa6380f51a0 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.167+0000 7fa636d9d700 1 --2- 172.21.15.173:0/2754586082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6380f51a0 0x7fa6381065d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fa6280089f0 tx=0x7fa62800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.167+0000 7fa61f7fe700 1 -- 172.21.15.173:0/2754586082 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa628009320 con 0x7fa6380f51a0 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.167+0000 7fa63c82f700 1 -- 172.21.15.173:0/2754586082 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa638106cc0 con 0x7fa6380f51a0 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.167+0000 7fa61f7fe700 1 -- 172.21.15.173:0/2754586082 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa628013380 con 0x7fa6380f51a0 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.167+0000 7fa61f7fe700 1 -- 172.21.15.173:0/2754586082 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa628014e20 con 0x7fa6380f51a0 2024-02-19T22:00:25.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.167+0000 7fa63c82f700 1 -- 172.21.15.173:0/2754586082 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa638107150 con 0x7fa6380f51a0 2024-02-19T22:00:25.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.168+0000 7fa63c82f700 1 -- 172.21.15.173:0/2754586082 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa638045ad0 con 0x7fa6380f51a0 2024-02-19T22:00:25.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.168+0000 7fa61f7fe700 1 -- 172.21.15.173:0/2754586082 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6280095f0 con 0x7fa6380f51a0 2024-02-19T22:00:25.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.168+0000 7fa61f7fe700 1 --2- 172.21.15.173:0/2754586082 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa62002ff10 0x7fa6200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:25.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.168+0000 7fa61f7fe700 1 -- 172.21.15.173:0/2754586082 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa628040ec0 con 0x7fa6380f51a0 2024-02-19T22:00:25.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.169+0000 7fa63659c700 1 --2- 172.21.15.173:0/2754586082 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa62002ff10 0x7fa6200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:25.170 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.169+0000 7fa63659c700 1 --2- 172.21.15.173:0/2754586082 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa62002ff10 0x7fa6200323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa62c0096a0 tx=0x7fa62c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:25.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.173+0000 7fa61f7fe700 1 -- 172.21.15.173:0/2754586082 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa628014420 con 0x7fa6380f51a0 2024-02-19T22:00:25.381 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.381+0000 7fa63c82f700 1 -- 172.21.15.173:0/2754586082 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa638108770 con 0x7fa6380f51a0 2024-02-19T22:00:25.381 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.382+0000 7fa61f7fe700 1 -- 172.21.15.173:0/2754586082 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa628010230 con 0x7fa6380f51a0 2024-02-19T22:00:25.382 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:25.382 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:25.383 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.383+0000 7fa63c82f700 1 -- 172.21.15.173:0/2754586082 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa62002ff10 msgr2=0x7fa6200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:25.383 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.383+0000 7fa63c82f700 1 --2- 172.21.15.173:0/2754586082 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa62002ff10 0x7fa6200323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa62c0096a0 tx=0x7fa62c006b40).stop 2024-02-19T22:00:25.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.383+0000 7fa63c82f700 1 -- 172.21.15.173:0/2754586082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6380f51a0 msgr2=0x7fa6381065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:25.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.383+0000 7fa63c82f700 1 --2- 172.21.15.173:0/2754586082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6380f51a0 0x7fa6381065d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fa6280089f0 tx=0x7fa62800b040).stop 2024-02-19T22:00:25.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.383+0000 7fa63c82f700 1 -- 172.21.15.173:0/2754586082 shutdown_connections 2024-02-19T22:00:25.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.383+0000 7fa63c82f700 1 --2- 172.21.15.173:0/2754586082 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa62002ff10 0x7fa6200323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:25.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.383+0000 7fa63c82f700 1 --2- 172.21.15.173:0/2754586082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6380f51a0 0x7fa6381065d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:25.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.383+0000 7fa63c82f700 1 -- 172.21.15.173:0/2754586082 >> 172.21.15.173:0/2754586082 conn(0x7fa6380f09a0 msgr2=0x7fa63805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:25.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.383+0000 7fa63c82f700 1 -- 172.21.15.173:0/2754586082 shutdown_connections 2024-02-19T22:00:25.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:25.383+0000 7fa63c82f700 1 -- 172.21.15.173:0/2754586082 wait complete. 2024-02-19T22:00:25.384 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:25.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:25 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:25.382892+0000 mon.smithi143 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.173:0/2754586082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:26.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:24.493229+0000 mgr.smithi143.bissbc (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:27.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:27.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:26.493608+0000 mgr.smithi143.bissbc (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:29.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.190+0000 7faaf261e700 1 -- 172.21.15.173:0/3011500593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaec0f5410 msgr2=0x7faaec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:29.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.190+0000 7faaf261e700 1 --2- 172.21.15.173:0/3011500593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaec0f5410 0x7faaec0f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7faae0009fb0 tx=0x7faae0009b30).stop 2024-02-19T22:00:29.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.190+0000 7faaf261e700 1 -- 172.21.15.173:0/3011500593 shutdown_connections 2024-02-19T22:00:29.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.190+0000 7faaf261e700 1 --2- 172.21.15.173:0/3011500593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaec0f5410 0x7faaec0f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:29.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.190+0000 7faaf261e700 1 -- 172.21.15.173:0/3011500593 >> 172.21.15.173:0/3011500593 conn(0x7faaec0f0b70 msgr2=0x7faaec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:29.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.190+0000 7faaf261e700 1 -- 172.21.15.173:0/3011500593 shutdown_connections 2024-02-19T22:00:29.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.190+0000 7faaf261e700 1 -- 172.21.15.173:0/3011500593 wait complete. 2024-02-19T22:00:29.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.191+0000 7faaf261e700 1 Processor -- start 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.191+0000 7faaf261e700 1 -- start start 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.191+0000 7faaf261e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaec0f5410 0x7faaec1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.191+0000 7faaf261e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec106cd0 con 0x7faaec0f5410 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.191+0000 7faaebfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaec0f5410 0x7faaec1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.191+0000 7faaebfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaec0f5410 0x7faaec1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:57502/0 (socket says 172.21.15.173:57502) 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.191+0000 7faaebfff700 1 -- 172.21.15.173:0/3430567897 learned_addr learned my addr 172.21.15.173:0/3430567897 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.191+0000 7faaebfff700 1 -- 172.21.15.173:0/3430567897 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae0005040 con 0x7faaec0f5410 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.191+0000 7faaebfff700 1 --2- 172.21.15.173:0/3430567897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaec0f5410 0x7faaec1067e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7faae0005010 tx=0x7faae000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.192+0000 7faae8ff9700 1 -- 172.21.15.173:0/3430567897 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faae0009320 con 0x7faaec0f5410 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.192+0000 7faaf261e700 1 -- 172.21.15.173:0/3430567897 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaec106ed0 con 0x7faaec0f5410 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.192+0000 7faae8ff9700 1 -- 172.21.15.173:0/3430567897 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faae0013380 con 0x7faaec0f5410 2024-02-19T22:00:29.193 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.192+0000 7faae8ff9700 1 -- 172.21.15.173:0/3430567897 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faae0014e50 con 0x7faaec0f5410 2024-02-19T22:00:29.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.192+0000 7faaf261e700 1 -- 172.21.15.173:0/3430567897 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaec107360 con 0x7faaec0f5410 2024-02-19T22:00:29.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.193+0000 7faae8ff9700 1 -- 172.21.15.173:0/3430567897 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faae00095f0 con 0x7faaec0f5410 2024-02-19T22:00:29.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.193+0000 7faaf261e700 1 -- 172.21.15.173:0/3430567897 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaec045ad0 con 0x7faaec0f5410 2024-02-19T22:00:29.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.193+0000 7faae8ff9700 1 --2- 172.21.15.173:0/3430567897 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faad402ff10 0x7faad40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:29.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.193+0000 7faae8ff9700 1 -- 172.21.15.173:0/3430567897 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faae0040e70 con 0x7faaec0f5410 2024-02-19T22:00:29.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.193+0000 7faaeb7fe700 1 --2- 172.21.15.173:0/3430567897 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faad402ff10 0x7faad40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:29.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.193+0000 7faaeb7fe700 1 --2- 172.21.15.173:0/3430567897 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faad402ff10 0x7faad40323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7faadc0096a0 tx=0x7faadc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:29.197 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.198+0000 7faae8ff9700 1 -- 172.21.15.173:0/3430567897 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faae0014420 con 0x7faaec0f5410 2024-02-19T22:00:29.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.404+0000 7faaf261e700 1 -- 172.21.15.173:0/3430567897 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faaec0fafa0 con 0x7faaec0f5410 2024-02-19T22:00:29.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.405+0000 7faae8ff9700 1 -- 172.21.15.173:0/3430567897 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faae0010230 con 0x7faaec0f5410 2024-02-19T22:00:29.404 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:29.405 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:29.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.406+0000 7faaf261e700 1 -- 172.21.15.173:0/3430567897 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faad402ff10 msgr2=0x7faad40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:29.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.406+0000 7faaf261e700 1 --2- 172.21.15.173:0/3430567897 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faad402ff10 0x7faad40323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7faadc0096a0 tx=0x7faadc006b40).stop 2024-02-19T22:00:29.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.406+0000 7faaf261e700 1 -- 172.21.15.173:0/3430567897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaec0f5410 msgr2=0x7faaec1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:29.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.406+0000 7faaf261e700 1 --2- 172.21.15.173:0/3430567897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaec0f5410 0x7faaec1067e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7faae0005010 tx=0x7faae000b040).stop 2024-02-19T22:00:29.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.406+0000 7faaf261e700 1 -- 172.21.15.173:0/3430567897 shutdown_connections 2024-02-19T22:00:29.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.406+0000 7faaf261e700 1 --2- 172.21.15.173:0/3430567897 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7faad402ff10 0x7faad40323d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:29.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.406+0000 7faaf261e700 1 --2- 172.21.15.173:0/3430567897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faaec0f5410 0x7faaec1067e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:29.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.406+0000 7faaf261e700 1 -- 172.21.15.173:0/3430567897 >> 172.21.15.173:0/3430567897 conn(0x7faaec0f0b70 msgr2=0x7faaec05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:29.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.406+0000 7faaf261e700 1 -- 172.21.15.173:0/3430567897 shutdown_connections 2024-02-19T22:00:29.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:29.406+0000 7faaf261e700 1 -- 172.21.15.173:0/3430567897 wait complete. 2024-02-19T22:00:29.407 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:29.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:29 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:29.405723+0000 mon.smithi143 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.173:0/3430567897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:28.493908+0000 mgr.smithi143.bissbc (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:31.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:31.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:32.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:30.494123+0000 mgr.smithi143.bissbc (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:33.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.359+0000 7f4cae106700 1 -- 172.21.15.173:0/542609081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca80f5410 msgr2=0x7f4ca80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:33.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.359+0000 7f4cae106700 1 --2- 172.21.15.173:0/542609081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca80f5410 0x7f4ca80f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f4c98009fb0 tx=0x7f4c98009b30).stop 2024-02-19T22:00:33.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.359+0000 7f4cae106700 1 -- 172.21.15.173:0/542609081 shutdown_connections 2024-02-19T22:00:33.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.359+0000 7f4cae106700 1 --2- 172.21.15.173:0/542609081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca80f5410 0x7f4ca80f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:33.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.359+0000 7f4cae106700 1 -- 172.21.15.173:0/542609081 >> 172.21.15.173:0/542609081 conn(0x7f4ca80f0b70 msgr2=0x7f4ca80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:33.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.359+0000 7f4cae106700 1 -- 172.21.15.173:0/542609081 shutdown_connections 2024-02-19T22:00:33.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.359+0000 7f4cae106700 1 -- 172.21.15.173:0/542609081 wait complete. 2024-02-19T22:00:33.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.360+0000 7f4cae106700 1 Processor -- start 2024-02-19T22:00:33.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.360+0000 7f4cae106700 1 -- start start 2024-02-19T22:00:33.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.360+0000 7f4cae106700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca80f5410 0x7f4ca8105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.360+0000 7f4cae106700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca8105c10 con 0x7f4ca80f5410 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.360+0000 7f4ca77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca80f5410 0x7f4ca8105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.360+0000 7f4ca77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca80f5410 0x7f4ca8105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:33440/0 (socket says 172.21.15.173:33440) 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.360+0000 7f4ca77fe700 1 -- 172.21.15.173:0/4168961438 learned_addr learned my addr 172.21.15.173:0/4168961438 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.361+0000 7f4ca77fe700 1 -- 172.21.15.173:0/4168961438 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c98005040 con 0x7f4ca80f5410 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.361+0000 7f4ca77fe700 1 --2- 172.21.15.173:0/4168961438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca80f5410 0x7f4ca8105720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f4c98005010 tx=0x7f4c9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.361+0000 7f4c9ffff700 1 -- 172.21.15.173:0/4168961438 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c98009320 con 0x7f4ca80f5410 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.361+0000 7f4cae106700 1 -- 172.21.15.173:0/4168961438 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca8105e10 con 0x7f4ca80f5410 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.361+0000 7f4c9ffff700 1 -- 172.21.15.173:0/4168961438 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c98013380 con 0x7f4ca80f5410 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.361+0000 7f4c9ffff700 1 -- 172.21.15.173:0/4168961438 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c98014e50 con 0x7f4ca80f5410 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.361+0000 7f4cae106700 1 -- 172.21.15.173:0/4168961438 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca81062a0 con 0x7f4ca80f5410 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.362+0000 7f4c9ffff700 1 -- 172.21.15.173:0/4168961438 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c980095f0 con 0x7f4ca80f5410 2024-02-19T22:00:33.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.362+0000 7f4cae106700 1 -- 172.21.15.173:0/4168961438 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca8045ad0 con 0x7f4ca80f5410 2024-02-19T22:00:33.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.362+0000 7f4c9ffff700 1 --2- 172.21.15.173:0/4168961438 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4c9402ff10 0x7f4c940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:33.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.362+0000 7f4c9ffff700 1 -- 172.21.15.173:0/4168961438 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c98040e90 con 0x7f4ca80f5410 2024-02-19T22:00:33.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.362+0000 7f4ca6ffd700 1 --2- 172.21.15.173:0/4168961438 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4c9402ff10 0x7f4c940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:33.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.363+0000 7f4ca6ffd700 1 --2- 172.21.15.173:0/4168961438 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4c9402ff10 0x7f4c940323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f4c900096a0 tx=0x7f4c90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:33.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.367+0000 7f4c9ffff700 1 -- 172.21.15.173:0/4168961438 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c98014420 con 0x7f4ca80f5410 2024-02-19T22:00:33.576 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.576+0000 7f4cae106700 1 -- 172.21.15.173:0/4168961438 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ca81070f0 con 0x7f4ca80f5410 2024-02-19T22:00:33.576 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.577+0000 7f4c9ffff700 1 -- 172.21.15.173:0/4168961438 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4c98010230 con 0x7f4ca80f5410 2024-02-19T22:00:33.576 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:33.576 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:33.578 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.578+0000 7f4cae106700 1 -- 172.21.15.173:0/4168961438 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4c9402ff10 msgr2=0x7f4c940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:33.578 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.578+0000 7f4cae106700 1 --2- 172.21.15.173:0/4168961438 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4c9402ff10 0x7f4c940323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f4c900096a0 tx=0x7f4c90006b40).stop 2024-02-19T22:00:33.578 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.578+0000 7f4cae106700 1 -- 172.21.15.173:0/4168961438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca80f5410 msgr2=0x7f4ca8105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:33.578 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.578+0000 7f4cae106700 1 --2- 172.21.15.173:0/4168961438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca80f5410 0x7f4ca8105720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f4c98005010 tx=0x7f4c9800b040).stop 2024-02-19T22:00:33.578 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.578+0000 7f4cae106700 1 -- 172.21.15.173:0/4168961438 shutdown_connections 2024-02-19T22:00:33.578 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.578+0000 7f4cae106700 1 --2- 172.21.15.173:0/4168961438 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4c9402ff10 0x7f4c940323d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:33.578 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.578+0000 7f4cae106700 1 --2- 172.21.15.173:0/4168961438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca80f5410 0x7f4ca8105720 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:33.578 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.578+0000 7f4cae106700 1 -- 172.21.15.173:0/4168961438 >> 172.21.15.173:0/4168961438 conn(0x7f4ca80f0b70 msgr2=0x7f4ca80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:33.578 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.578+0000 7f4cae106700 1 -- 172.21.15.173:0/4168961438 shutdown_connections 2024-02-19T22:00:33.579 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:33.578+0000 7f4cae106700 1 -- 172.21.15.173:0/4168961438 wait complete. 2024-02-19T22:00:33.579 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:32.494355+0000 mgr.smithi143.bissbc (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:33.577885+0000 mon.smithi143 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.173:0/4168961438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:35.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:35.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:34.494578+0000 mgr.smithi143.bissbc (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:37.384 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.383+0000 7f3529a10700 1 -- 172.21.15.173:0/386308506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35240f6510 msgr2=0x7f35240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:37.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.383+0000 7f3529a10700 1 --2- 172.21.15.173:0/386308506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35240f6510 0x7f35240f6930 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f3514009fb0 tx=0x7f3514009b30).stop 2024-02-19T22:00:37.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.383+0000 7f3529a10700 1 -- 172.21.15.173:0/386308506 shutdown_connections 2024-02-19T22:00:37.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.383+0000 7f3529a10700 1 --2- 172.21.15.173:0/386308506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35240f6510 0x7f35240f6930 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:37.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.383+0000 7f3529a10700 1 -- 172.21.15.173:0/386308506 >> 172.21.15.173:0/386308506 conn(0x7f35240f1c70 msgr2=0x7f35240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:37.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.383+0000 7f3529a10700 1 -- 172.21.15.173:0/386308506 shutdown_connections 2024-02-19T22:00:37.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.383+0000 7f3529a10700 1 -- 172.21.15.173:0/386308506 wait complete. 2024-02-19T22:00:37.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.384+0000 7f3529a10700 1 Processor -- start 2024-02-19T22:00:37.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.384+0000 7f3529a10700 1 -- start start 2024-02-19T22:00:37.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.384+0000 7f3529a10700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35240f6510 0x7f35241067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:37.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.384+0000 7f3529a10700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3524106cd0 con 0x7f35240f6510 2024-02-19T22:00:37.385 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.384+0000 7f3522ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35240f6510 0x7f35241067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:37.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.384+0000 7f3522ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35240f6510 0x7f35241067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:33456/0 (socket says 172.21.15.173:33456) 2024-02-19T22:00:37.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.384+0000 7f3522ffd700 1 -- 172.21.15.173:0/2882704811 learned_addr learned my addr 172.21.15.173:0/2882704811 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:37.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.385+0000 7f3522ffd700 1 -- 172.21.15.173:0/2882704811 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3514005040 con 0x7f35240f6510 2024-02-19T22:00:37.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.385+0000 7f3522ffd700 1 --2- 172.21.15.173:0/2882704811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35240f6510 0x7f35241067e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f3514005010 tx=0x7f351400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:37.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.385+0000 7f351b7fe700 1 -- 172.21.15.173:0/2882704811 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3514009320 con 0x7f35240f6510 2024-02-19T22:00:37.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.385+0000 7f3529a10700 1 -- 172.21.15.173:0/2882704811 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3524106ed0 con 0x7f35240f6510 2024-02-19T22:00:37.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.385+0000 7f351b7fe700 1 -- 172.21.15.173:0/2882704811 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3514013380 con 0x7f35240f6510 2024-02-19T22:00:37.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.385+0000 7f351b7fe700 1 -- 172.21.15.173:0/2882704811 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3514014e50 con 0x7f35240f6510 2024-02-19T22:00:37.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.385+0000 7f3529a10700 1 -- 172.21.15.173:0/2882704811 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3524107360 con 0x7f35240f6510 2024-02-19T22:00:37.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.386+0000 7f351b7fe700 1 -- 172.21.15.173:0/2882704811 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35140263a0 con 0x7f35240f6510 2024-02-19T22:00:37.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.386+0000 7f3529a10700 1 -- 172.21.15.173:0/2882704811 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3524045ad0 con 0x7f35240f6510 2024-02-19T22:00:37.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.386+0000 7f351b7fe700 1 --2- 172.21.15.173:0/2882704811 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f351002ff10 0x7f35100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:37.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.386+0000 7f351b7fe700 1 -- 172.21.15.173:0/2882704811 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3514040f50 con 0x7f35240f6510 2024-02-19T22:00:37.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.386+0000 7f35227fc700 1 --2- 172.21.15.173:0/2882704811 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f351002ff10 0x7f35100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:37.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.386+0000 7f35227fc700 1 --2- 172.21.15.173:0/2882704811 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f351002ff10 0x7f35100323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f350c0096a0 tx=0x7f350c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:37.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.391+0000 7f351b7fe700 1 -- 172.21.15.173:0/2882704811 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3514012330 con 0x7f35240f6510 2024-02-19T22:00:37.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.595+0000 7f3529a10700 1 -- 172.21.15.173:0/2882704811 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3524108190 con 0x7f35240f6510 2024-02-19T22:00:37.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.596+0000 7f351b7fe700 1 -- 172.21.15.173:0/2882704811 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f351401db70 con 0x7f35240f6510 2024-02-19T22:00:37.595 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:37.595 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:37.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.597+0000 7f3529a10700 1 -- 172.21.15.173:0/2882704811 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f351002ff10 msgr2=0x7f35100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:37.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.597+0000 7f3529a10700 1 --2- 172.21.15.173:0/2882704811 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f351002ff10 0x7f35100323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f350c0096a0 tx=0x7f350c006b40).stop 2024-02-19T22:00:37.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.597+0000 7f3529a10700 1 -- 172.21.15.173:0/2882704811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35240f6510 msgr2=0x7f35241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:37.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.597+0000 7f3529a10700 1 --2- 172.21.15.173:0/2882704811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35240f6510 0x7f35241067e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f3514005010 tx=0x7f351400b040).stop 2024-02-19T22:00:37.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.597+0000 7f3529a10700 1 -- 172.21.15.173:0/2882704811 shutdown_connections 2024-02-19T22:00:37.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.597+0000 7f3529a10700 1 --2- 172.21.15.173:0/2882704811 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f351002ff10 0x7f35100323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:37.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.597+0000 7f3529a10700 1 --2- 172.21.15.173:0/2882704811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35240f6510 0x7f35241067e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:37.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.597+0000 7f3529a10700 1 -- 172.21.15.173:0/2882704811 >> 172.21.15.173:0/2882704811 conn(0x7f35240f1c70 msgr2=0x7f35240f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:37.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.597+0000 7f3529a10700 1 -- 172.21.15.173:0/2882704811 shutdown_connections 2024-02-19T22:00:37.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:37.597+0000 7f3529a10700 1 -- 172.21.15.173:0/2882704811 wait complete. 2024-02-19T22:00:37.598 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:36.494823+0000 mgr.smithi143.bissbc (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:37.596816+0000 mon.smithi143 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.173:0/2882704811' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:39.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:39.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:38.495065+0000 mgr.smithi143.bissbc (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:41.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.300+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1136708711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c00f5410 msgr2=0x7f41c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:41.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.300+0000 7f41c4b2a700 1 --2- 172.21.15.173:0/1136708711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c00f5410 0x7f41c00f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f41b8009fb0 tx=0x7f41b8009b30).stop 2024-02-19T22:00:41.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.300+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1136708711 shutdown_connections 2024-02-19T22:00:41.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.300+0000 7f41c4b2a700 1 --2- 172.21.15.173:0/1136708711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c00f5410 0x7f41c00f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:41.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.300+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1136708711 >> 172.21.15.173:0/1136708711 conn(0x7f41c00f0b70 msgr2=0x7f41c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:41.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.300+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1136708711 shutdown_connections 2024-02-19T22:00:41.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.300+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1136708711 wait complete. 2024-02-19T22:00:41.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.301+0000 7f41c4b2a700 1 Processor -- start 2024-02-19T22:00:41.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.301+0000 7f41c4b2a700 1 -- start start 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.301+0000 7f41c4b2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c00f5410 0x7f41c01094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.301+0000 7f41c4b2a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c0060280 con 0x7f41c00f5410 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.301+0000 7f41be59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c00f5410 0x7f41c01094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.301+0000 7f41be59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c00f5410 0x7f41c01094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:39056/0 (socket says 172.21.15.173:39056) 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.301+0000 7f41be59c700 1 -- 172.21.15.173:0/1084939728 learned_addr learned my addr 172.21.15.173:0/1084939728 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.301+0000 7f41be59c700 1 -- 172.21.15.173:0/1084939728 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b8005040 con 0x7f41c00f5410 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.302+0000 7f41be59c700 1 --2- 172.21.15.173:0/1084939728 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c00f5410 0x7f41c01094a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f41b8005010 tx=0x7f41b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.302+0000 7f41b6ffd700 1 -- 172.21.15.173:0/1084939728 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b8009320 con 0x7f41c00f5410 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.302+0000 7f41b6ffd700 1 -- 172.21.15.173:0/1084939728 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41b8013380 con 0x7f41c00f5410 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.302+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1084939728 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c0060480 con 0x7f41c00f5410 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.302+0000 7f41b6ffd700 1 -- 172.21.15.173:0/1084939728 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41b8014e50 con 0x7f41c00f5410 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.302+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1084939728 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c0109bd0 con 0x7f41c00f5410 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.303+0000 7f41b6ffd700 1 -- 172.21.15.173:0/1084939728 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41b80095f0 con 0x7f41c00f5410 2024-02-19T22:00:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.303+0000 7f41b6ffd700 1 --2- 172.21.15.173:0/1084939728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41ac02ff10 0x7f41ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:41.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.303+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1084939728 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c0045ad0 con 0x7f41c00f5410 2024-02-19T22:00:41.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.303+0000 7f41b6ffd700 1 -- 172.21.15.173:0/1084939728 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41b8040fa0 con 0x7f41c00f5410 2024-02-19T22:00:41.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.303+0000 7f41bdd9b700 1 --2- 172.21.15.173:0/1084939728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41ac02ff10 0x7f41ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:41.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.304+0000 7f41bdd9b700 1 --2- 172.21.15.173:0/1084939728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41ac02ff10 0x7f41ac0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f41a80096a0 tx=0x7f41a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:41.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.308+0000 7f41b6ffd700 1 -- 172.21.15.173:0/1084939728 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41b801dbc0 con 0x7f41c00f5410 2024-02-19T22:00:41.511 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.512+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1084939728 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41c00fafa0 con 0x7f41c00f5410 2024-02-19T22:00:41.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.513+0000 7f41b6ffd700 1 -- 172.21.15.173:0/1084939728 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f41b8010230 con 0x7f41c00f5410 2024-02-19T22:00:41.512 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:41.513 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:41.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.514+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1084939728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41ac02ff10 msgr2=0x7f41ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:41.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.514+0000 7f41c4b2a700 1 --2- 172.21.15.173:0/1084939728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41ac02ff10 0x7f41ac0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f41a80096a0 tx=0x7f41a8006b40).stop 2024-02-19T22:00:41.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.514+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1084939728 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c00f5410 msgr2=0x7f41c01094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:41.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.514+0000 7f41c4b2a700 1 --2- 172.21.15.173:0/1084939728 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c00f5410 0x7f41c01094a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f41b8005010 tx=0x7f41b800b040).stop 2024-02-19T22:00:41.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.514+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1084939728 shutdown_connections 2024-02-19T22:00:41.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.514+0000 7f41c4b2a700 1 --2- 172.21.15.173:0/1084939728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41ac02ff10 0x7f41ac0323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:41.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.514+0000 7f41c4b2a700 1 --2- 172.21.15.173:0/1084939728 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41c00f5410 0x7f41c01094a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:41.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.514+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1084939728 >> 172.21.15.173:0/1084939728 conn(0x7f41c00f0b70 msgr2=0x7f41c00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:41.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.514+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1084939728 shutdown_connections 2024-02-19T22:00:41.515 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:41.514+0000 7f41c4b2a700 1 -- 172.21.15.173:0/1084939728 wait complete. 2024-02-19T22:00:41.515 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:40.495327+0000 mgr.smithi143.bissbc (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:41.513956+0000 mon.smithi143 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.173:0/1084939728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:43.309 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:43.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:42.495712+0000 mgr.smithi143.bissbc (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:45.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.409+0000 7f72a6156700 1 -- 172.21.15.173:0/2259229645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a00f51a0 msgr2=0x7f72a00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:45.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.409+0000 7f72a6156700 1 --2- 172.21.15.173:0/2259229645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a00f51a0 0x7f72a00f55c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f7290009fb0 tx=0x7f7290009b30).stop 2024-02-19T22:00:45.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.409+0000 7f72a6156700 1 -- 172.21.15.173:0/2259229645 shutdown_connections 2024-02-19T22:00:45.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.409+0000 7f72a6156700 1 --2- 172.21.15.173:0/2259229645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a00f51a0 0x7f72a00f55c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:45.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.409+0000 7f72a6156700 1 -- 172.21.15.173:0/2259229645 >> 172.21.15.173:0/2259229645 conn(0x7f72a00f09a0 msgr2=0x7f72a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:45.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.409+0000 7f72a6156700 1 -- 172.21.15.173:0/2259229645 shutdown_connections 2024-02-19T22:00:45.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.409+0000 7f72a6156700 1 -- 172.21.15.173:0/2259229645 wait complete. 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.409+0000 7f72a6156700 1 Processor -- start 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.409+0000 7f72a6156700 1 -- start start 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.409+0000 7f72a6156700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a00f51a0 0x7f72a01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.409+0000 7f72a6156700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a0106ac0 con 0x7f72a00f51a0 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.410+0000 7f72a5154700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a00f51a0 0x7f72a01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.410+0000 7f72a5154700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a00f51a0 0x7f72a01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:39084/0 (socket says 172.21.15.173:39084) 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.410+0000 7f72a5154700 1 -- 172.21.15.173:0/1145807698 learned_addr learned my addr 172.21.15.173:0/1145807698 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.410+0000 7f72a5154700 1 -- 172.21.15.173:0/1145807698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7290005040 con 0x7f72a00f51a0 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.410+0000 7f72a5154700 1 --2- 172.21.15.173:0/1145807698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a00f51a0 0x7f72a01065d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f7290009830 tx=0x7f729000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.410+0000 7f7295ffb700 1 -- 172.21.15.173:0/1145807698 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7290009320 con 0x7f72a00f51a0 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.410+0000 7f72a6156700 1 -- 172.21.15.173:0/1145807698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a0106cc0 con 0x7f72a00f51a0 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.410+0000 7f7295ffb700 1 -- 172.21.15.173:0/1145807698 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7290013380 con 0x7f72a00f51a0 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.410+0000 7f7295ffb700 1 -- 172.21.15.173:0/1145807698 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7290014420 con 0x7f72a00f51a0 2024-02-19T22:00:45.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.410+0000 7f72a6156700 1 -- 172.21.15.173:0/1145807698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a0107150 con 0x7f72a00f51a0 2024-02-19T22:00:45.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.411+0000 7f7295ffb700 1 -- 172.21.15.173:0/1145807698 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f729001e420 con 0x7f72a00f51a0 2024-02-19T22:00:45.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.411+0000 7f72a6156700 1 -- 172.21.15.173:0/1145807698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a0045ad0 con 0x7f72a00f51a0 2024-02-19T22:00:45.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.412+0000 7f7295ffb700 1 --2- 172.21.15.173:0/1145807698 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f728c02ffb0 0x7f728c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:45.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.412+0000 7f7295ffb700 1 -- 172.21.15.173:0/1145807698 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7290040ef0 con 0x7f72a00f51a0 2024-02-19T22:00:45.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.412+0000 7f72a4953700 1 --2- 172.21.15.173:0/1145807698 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f728c02ffb0 0x7f728c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:45.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.412+0000 7f72a4953700 1 --2- 172.21.15.173:0/1145807698 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f728c02ffb0 0x7f728c032470 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f729c0096a0 tx=0x7f729c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:45.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.416+0000 7f7295ffb700 1 -- 172.21.15.173:0/1145807698 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f729001cb10 con 0x7f72a00f51a0 2024-02-19T22:00:45.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.625+0000 7f72a6156700 1 -- 172.21.15.173:0/1145807698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72a00fad30 con 0x7f72a00f51a0 2024-02-19T22:00:45.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.625+0000 7f7295ffb700 1 -- 172.21.15.173:0/1145807698 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7290004490 con 0x7f72a00f51a0 2024-02-19T22:00:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:45.625 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:45.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.627+0000 7f72a6156700 1 -- 172.21.15.173:0/1145807698 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f728c02ffb0 msgr2=0x7f728c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:45.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.627+0000 7f72a6156700 1 --2- 172.21.15.173:0/1145807698 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f728c02ffb0 0x7f728c032470 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f729c0096a0 tx=0x7f729c006b40).stop 2024-02-19T22:00:45.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.627+0000 7f72a6156700 1 -- 172.21.15.173:0/1145807698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a00f51a0 msgr2=0x7f72a01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:45.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.627+0000 7f72a6156700 1 --2- 172.21.15.173:0/1145807698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a00f51a0 0x7f72a01065d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f7290009830 tx=0x7f729000b040).stop 2024-02-19T22:00:45.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.627+0000 7f72a6156700 1 -- 172.21.15.173:0/1145807698 shutdown_connections 2024-02-19T22:00:45.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.627+0000 7f72a6156700 1 --2- 172.21.15.173:0/1145807698 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f728c02ffb0 0x7f728c032470 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:45.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.627+0000 7f72a6156700 1 --2- 172.21.15.173:0/1145807698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a00f51a0 0x7f72a01065d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:45.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.627+0000 7f72a6156700 1 -- 172.21.15.173:0/1145807698 >> 172.21.15.173:0/1145807698 conn(0x7f72a00f09a0 msgr2=0x7f72a005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:45.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.627+0000 7f72a6156700 1 -- 172.21.15.173:0/1145807698 shutdown_connections 2024-02-19T22:00:45.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:45.627+0000 7f72a6156700 1 -- 172.21.15.173:0/1145807698 wait complete. 2024-02-19T22:00:45.628 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:44.496123+0000 mgr.smithi143.bissbc (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:45.626481+0000 mon.smithi143 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.173:0/1145807698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:47.262 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:47.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:46.496443+0000 mgr.smithi143.bissbc (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:49.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.344+0000 7f83533c0700 1 -- 172.21.15.173:0/1379617885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f834c0f5410 msgr2=0x7f834c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:49.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.344+0000 7f83533c0700 1 --2- 172.21.15.173:0/1379617885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f834c0f5410 0x7f834c0f5830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f8348009fb0 tx=0x7f8348009b30).stop 2024-02-19T22:00:49.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.344+0000 7f83533c0700 1 -- 172.21.15.173:0/1379617885 shutdown_connections 2024-02-19T22:00:49.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.344+0000 7f83533c0700 1 --2- 172.21.15.173:0/1379617885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f834c0f5410 0x7f834c0f5830 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:49.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.344+0000 7f83533c0700 1 -- 172.21.15.173:0/1379617885 >> 172.21.15.173:0/1379617885 conn(0x7f834c0f0b70 msgr2=0x7f834c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:49.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.344+0000 7f83533c0700 1 -- 172.21.15.173:0/1379617885 shutdown_connections 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.344+0000 7f83533c0700 1 -- 172.21.15.173:0/1379617885 wait complete. 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.345+0000 7f83533c0700 1 Processor -- start 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.345+0000 7f83533c0700 1 -- start start 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.345+0000 7f83533c0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f834c0f5410 0x7f834c1094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.345+0000 7f83533c0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c060280 con 0x7f834c0f5410 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.345+0000 7f835115c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f834c0f5410 0x7f834c1094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.345+0000 7f835115c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f834c0f5410 0x7f834c1094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:39104/0 (socket says 172.21.15.173:39104) 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.345+0000 7f835115c700 1 -- 172.21.15.173:0/4027436888 learned_addr learned my addr 172.21.15.173:0/4027436888 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.345+0000 7f835115c700 1 -- 172.21.15.173:0/4027436888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8348005040 con 0x7f834c0f5410 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.345+0000 7f835115c700 1 --2- 172.21.15.173:0/4027436888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f834c0f5410 0x7f834c1094a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f8348005010 tx=0x7f834800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.346+0000 7f8341ffb700 1 -- 172.21.15.173:0/4027436888 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8348009320 con 0x7f834c0f5410 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.346+0000 7f8341ffb700 1 -- 172.21.15.173:0/4027436888 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8348013380 con 0x7f834c0f5410 2024-02-19T22:00:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.346+0000 7f83533c0700 1 -- 172.21.15.173:0/4027436888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834c060480 con 0x7f834c0f5410 2024-02-19T22:00:49.347 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.346+0000 7f8341ffb700 1 -- 172.21.15.173:0/4027436888 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8348014e50 con 0x7f834c0f5410 2024-02-19T22:00:49.347 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.346+0000 7f83533c0700 1 -- 172.21.15.173:0/4027436888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834c109bd0 con 0x7f834c0f5410 2024-02-19T22:00:49.347 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.347+0000 7f83533c0700 1 -- 172.21.15.173:0/4027436888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834c045ad0 con 0x7f834c0f5410 2024-02-19T22:00:49.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.347+0000 7f8341ffb700 1 -- 172.21.15.173:0/4027436888 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83480095f0 con 0x7f834c0f5410 2024-02-19T22:00:49.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.347+0000 7f8341ffb700 1 --2- 172.21.15.173:0/4027436888 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f833802ff10 0x7f83380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:49.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.347+0000 7f8341ffb700 1 -- 172.21.15.173:0/4027436888 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8348040e90 con 0x7f834c0f5410 2024-02-19T22:00:49.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.347+0000 7f835095b700 1 --2- 172.21.15.173:0/4027436888 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f833802ff10 0x7f83380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:49.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.348+0000 7f835095b700 1 --2- 172.21.15.173:0/4027436888 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f833802ff10 0x7f83380323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f833c0096a0 tx=0x7f833c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:49.351 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.352+0000 7f8341ffb700 1 -- 172.21.15.173:0/4027436888 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8348014420 con 0x7f834c0f5410 2024-02-19T22:00:49.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.561+0000 7f83533c0700 1 -- 172.21.15.173:0/4027436888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f834c0fafa0 con 0x7f834c0f5410 2024-02-19T22:00:49.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.561+0000 7f8341ffb700 1 -- 172.21.15.173:0/4027436888 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8348010230 con 0x7f834c0f5410 2024-02-19T22:00:49.561 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:49.562 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:49.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.563+0000 7f83533c0700 1 -- 172.21.15.173:0/4027436888 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f833802ff10 msgr2=0x7f83380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:49.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.563+0000 7f83533c0700 1 --2- 172.21.15.173:0/4027436888 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f833802ff10 0x7f83380323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f833c0096a0 tx=0x7f833c006b40).stop 2024-02-19T22:00:49.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.563+0000 7f83533c0700 1 -- 172.21.15.173:0/4027436888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f834c0f5410 msgr2=0x7f834c1094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:49.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.563+0000 7f83533c0700 1 --2- 172.21.15.173:0/4027436888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f834c0f5410 0x7f834c1094a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f8348005010 tx=0x7f834800b040).stop 2024-02-19T22:00:49.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.563+0000 7f83533c0700 1 -- 172.21.15.173:0/4027436888 shutdown_connections 2024-02-19T22:00:49.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.563+0000 7f83533c0700 1 --2- 172.21.15.173:0/4027436888 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f833802ff10 0x7f83380323d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:49.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.563+0000 7f83533c0700 1 --2- 172.21.15.173:0/4027436888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f834c0f5410 0x7f834c1094a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:49.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.563+0000 7f83533c0700 1 -- 172.21.15.173:0/4027436888 >> 172.21.15.173:0/4027436888 conn(0x7f834c0f0b70 msgr2=0x7f834c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:49.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.563+0000 7f83533c0700 1 -- 172.21.15.173:0/4027436888 shutdown_connections 2024-02-19T22:00:49.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:49.563+0000 7f83533c0700 1 -- 172.21.15.173:0/4027436888 wait complete. 2024-02-19T22:00:49.563 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:48.496811+0000 mgr.smithi143.bissbc (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:49.562507+0000 mon.smithi143 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.173:0/4027436888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:51.243 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:51.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:52.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:50.497050+0000 mgr.smithi143.bissbc (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:53.236 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.236+0000 7f46d2e40700 1 -- 172.21.15.173:0/489557592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46cc0f5410 msgr2=0x7f46cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.236+0000 7f46d2e40700 1 --2- 172.21.15.173:0/489557592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46cc0f5410 0x7f46cc0f5830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f46c0009fb0 tx=0x7f46c0009b30).stop 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.236+0000 7f46d2e40700 1 -- 172.21.15.173:0/489557592 shutdown_connections 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.236+0000 7f46d2e40700 1 --2- 172.21.15.173:0/489557592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46cc0f5410 0x7f46cc0f5830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.236+0000 7f46d2e40700 1 -- 172.21.15.173:0/489557592 >> 172.21.15.173:0/489557592 conn(0x7f46cc0f0b70 msgr2=0x7f46cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.236+0000 7f46d2e40700 1 -- 172.21.15.173:0/489557592 shutdown_connections 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.236+0000 7f46d2e40700 1 -- 172.21.15.173:0/489557592 wait complete. 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.236+0000 7f46d2e40700 1 Processor -- start 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.237+0000 7f46d2e40700 1 -- start start 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.237+0000 7f46d2e40700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46cc0f5410 0x7f46cc1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.237+0000 7f46d2e40700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc060240 con 0x7f46cc0f5410 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.237+0000 7f46d0bdc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46cc0f5410 0x7f46cc1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.237+0000 7f46d0bdc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46cc0f5410 0x7f46cc1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:33626/0 (socket says 172.21.15.173:33626) 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.237+0000 7f46d0bdc700 1 -- 172.21.15.173:0/3178557981 learned_addr learned my addr 172.21.15.173:0/3178557981 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.237+0000 7f46d0bdc700 1 -- 172.21.15.173:0/3178557981 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c0005040 con 0x7f46cc0f5410 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.237+0000 7f46d0bdc700 1 --2- 172.21.15.173:0/3178557981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46cc0f5410 0x7f46cc1094e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f46c0005010 tx=0x7f46c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.238+0000 7f46c97fa700 1 -- 172.21.15.173:0/3178557981 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46c0009320 con 0x7f46cc0f5410 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.238+0000 7f46d2e40700 1 -- 172.21.15.173:0/3178557981 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc060440 con 0x7f46cc0f5410 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.238+0000 7f46c97fa700 1 -- 172.21.15.173:0/3178557981 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46c0013380 con 0x7f46cc0f5410 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.238+0000 7f46c97fa700 1 -- 172.21.15.173:0/3178557981 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46c0014e50 con 0x7f46cc0f5410 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.238+0000 7f46d2e40700 1 -- 172.21.15.173:0/3178557981 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc109b90 con 0x7f46cc0f5410 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.238+0000 7f46c97fa700 1 -- 172.21.15.173:0/3178557981 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46c00263a0 con 0x7f46cc0f5410 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.239+0000 7f46c97fa700 1 --2- 172.21.15.173:0/3178557981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46bc02ff10 0x7f46bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.239+0000 7f46c97fa700 1 -- 172.21.15.173:0/3178557981 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46c0041210 con 0x7f46cc0f5410 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.239+0000 7f46d2e40700 1 -- 172.21.15.173:0/3178557981 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc045ad0 con 0x7f46cc0f5410 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.239+0000 7f46cbfff700 1 --2- 172.21.15.173:0/3178557981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46bc02ff10 0x7f46bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:53.239 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.239+0000 7f46cbfff700 1 --2- 172.21.15.173:0/3178557981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46bc02ff10 0x7f46bc0323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f46b80096a0 tx=0x7f46b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:53.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.244+0000 7f46c97fa700 1 -- 172.21.15.173:0/3178557981 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46c0012920 con 0x7f46cc0f5410 2024-02-19T22:00:53.449 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.450+0000 7f46d2e40700 1 -- 172.21.15.173:0/3178557981 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46cc10a3b0 con 0x7f46cc0f5410 2024-02-19T22:00:53.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.450+0000 7f46c97fa700 1 -- 172.21.15.173:0/3178557981 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f46c0012330 con 0x7f46cc0f5410 2024-02-19T22:00:53.450 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:53.450 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:53.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.452+0000 7f46d2e40700 1 -- 172.21.15.173:0/3178557981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46bc02ff10 msgr2=0x7f46bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:53.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.452+0000 7f46d2e40700 1 --2- 172.21.15.173:0/3178557981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46bc02ff10 0x7f46bc0323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f46b80096a0 tx=0x7f46b8006b40).stop 2024-02-19T22:00:53.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.452+0000 7f46d2e40700 1 -- 172.21.15.173:0/3178557981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46cc0f5410 msgr2=0x7f46cc1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:53.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.452+0000 7f46d2e40700 1 --2- 172.21.15.173:0/3178557981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46cc0f5410 0x7f46cc1094e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f46c0005010 tx=0x7f46c000b040).stop 2024-02-19T22:00:53.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.452+0000 7f46d2e40700 1 -- 172.21.15.173:0/3178557981 shutdown_connections 2024-02-19T22:00:53.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.452+0000 7f46d2e40700 1 --2- 172.21.15.173:0/3178557981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f46bc02ff10 0x7f46bc0323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:53.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.452+0000 7f46d2e40700 1 --2- 172.21.15.173:0/3178557981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46cc0f5410 0x7f46cc1094e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:53.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.452+0000 7f46d2e40700 1 -- 172.21.15.173:0/3178557981 >> 172.21.15.173:0/3178557981 conn(0x7f46cc0f0b70 msgr2=0x7f46cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:53.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.452+0000 7f46d2e40700 1 -- 172.21.15.173:0/3178557981 shutdown_connections 2024-02-19T22:00:53.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:53.452+0000 7f46d2e40700 1 -- 172.21.15.173:0/3178557981 wait complete. 2024-02-19T22:00:53.453 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:53.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:53 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:53.451602+0000 mon.smithi143 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.173:0/3178557981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:52.497339+0000 mgr.smithi143.bissbc (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:55.133 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:55.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:54.497656+0000 mgr.smithi143.bissbc (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:57.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.071+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/2524230618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df4060280 msgr2=0x7f0df40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.071+0000 7f0dfa1bc700 1 --2- 172.21.15.173:0/2524230618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df4060280 0x7f0df40f90f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f0ddc009fb0 tx=0x7f0ddc009b30).stop 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.071+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/2524230618 shutdown_connections 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.071+0000 7f0dfa1bc700 1 --2- 172.21.15.173:0/2524230618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df4060280 0x7f0df40f90f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.071+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/2524230618 >> 172.21.15.173:0/2524230618 conn(0x7f0df40f0b70 msgr2=0x7f0df40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.071+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/2524230618 shutdown_connections 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.071+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/2524230618 wait complete. 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.071+0000 7f0dfa1bc700 1 Processor -- start 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.071+0000 7f0dfa1bc700 1 -- start start 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.072+0000 7f0dfa1bc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df4060280 0x7f0df4106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.072+0000 7f0dfa1bc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df4106c70 con 0x7f0df4060280 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.072+0000 7f0df37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df4060280 0x7f0df4106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:57.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.072+0000 7f0df37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df4060280 0x7f0df4106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:33638/0 (socket says 172.21.15.173:33638) 2024-02-19T22:00:57.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.072+0000 7f0df37fe700 1 -- 172.21.15.173:0/1216712490 learned_addr learned my addr 172.21.15.173:0/1216712490 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:00:57.074 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.072+0000 7f0df37fe700 1 -- 172.21.15.173:0/1216712490 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc005040 con 0x7f0df4060280 2024-02-19T22:00:57.074 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.073+0000 7f0df37fe700 1 --2- 172.21.15.173:0/1216712490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df4060280 0x7f0df4106780 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f0ddc009830 tx=0x7f0ddc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:57.074 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.073+0000 7f0debfff700 1 -- 172.21.15.173:0/1216712490 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ddc009320 con 0x7f0df4060280 2024-02-19T22:00:57.074 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.073+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/1216712490 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df4106e70 con 0x7f0df4060280 2024-02-19T22:00:57.074 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.073+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/1216712490 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df4107300 con 0x7f0df4060280 2024-02-19T22:00:57.074 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.073+0000 7f0debfff700 1 -- 172.21.15.173:0/1216712490 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ddc013380 con 0x7f0df4060280 2024-02-19T22:00:57.074 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.073+0000 7f0debfff700 1 -- 172.21.15.173:0/1216712490 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ddc014420 con 0x7f0df4060280 2024-02-19T22:00:57.075 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.074+0000 7f0debfff700 1 -- 172.21.15.173:0/1216712490 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ddc014580 con 0x7f0df4060280 2024-02-19T22:00:57.075 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.074+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/1216712490 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd8004fa0 con 0x7f0df4060280 2024-02-19T22:00:57.075 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.074+0000 7f0debfff700 1 --2- 172.21.15.173:0/1216712490 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0de002ff60 0x7f0de0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:57.075 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.074+0000 7f0debfff700 1 -- 172.21.15.173:0/1216712490 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ddc0410c0 con 0x7f0df4060280 2024-02-19T22:00:57.075 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.074+0000 7f0df2ffd700 1 --2- 172.21.15.173:0/1216712490 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0de002ff60 0x7f0de0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:57.075 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.074+0000 7f0df2ffd700 1 --2- 172.21.15.173:0/1216712490 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0de002ff60 0x7f0de0032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0de40096a0 tx=0x7f0de4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:57.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.079+0000 7f0debfff700 1 -- 172.21.15.173:0/1216712490 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ddc01c6e0 con 0x7f0df4060280 2024-02-19T22:00:57.285 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.286+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/1216712490 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0dd8005670 con 0x7f0df4060280 2024-02-19T22:00:57.286 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.286+0000 7f0debfff700 1 -- 172.21.15.173:0/1216712490 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0ddc02c2d0 con 0x7f0df4060280 2024-02-19T22:00:57.286 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:00:57.286 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.288+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/1216712490 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0de002ff60 msgr2=0x7f0de0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.288+0000 7f0dfa1bc700 1 --2- 172.21.15.173:0/1216712490 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0de002ff60 0x7f0de0032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0de40096a0 tx=0x7f0de4006b40).stop 2024-02-19T22:00:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.288+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/1216712490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df4060280 msgr2=0x7f0df4106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.288+0000 7f0dfa1bc700 1 --2- 172.21.15.173:0/1216712490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df4060280 0x7f0df4106780 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f0ddc009830 tx=0x7f0ddc00b040).stop 2024-02-19T22:00:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.288+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/1216712490 shutdown_connections 2024-02-19T22:00:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.288+0000 7f0dfa1bc700 1 --2- 172.21.15.173:0/1216712490 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0de002ff60 0x7f0de0032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.288+0000 7f0dfa1bc700 1 --2- 172.21.15.173:0/1216712490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df4060280 0x7f0df4106780 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.288+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/1216712490 >> 172.21.15.173:0/1216712490 conn(0x7f0df40f0b70 msgr2=0x7f0df40f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.288+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/1216712490 shutdown_connections 2024-02-19T22:00:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:00:57.288+0000 7f0dfa1bc700 1 -- 172.21.15.173:0/1216712490 wait complete. 2024-02-19T22:00:57.289 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:00:57.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:57 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:00:57.287508+0000 mon.smithi143 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.173:0/1216712490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:58.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:58.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:00:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:56.498023+0000 mgr.smithi143.bissbc (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:00:58.498234+0000 mgr.smithi143.bissbc (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:00.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.924+0000 7f5117a34700 1 -- 172.21.15.173:0/4028152972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51100f5410 msgr2=0x7f51100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.924+0000 7f5117a34700 1 --2- 172.21.15.173:0/4028152972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51100f5410 0x7f51100f5830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f510c009fb0 tx=0x7f510c009b30).stop 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.924+0000 7f5117a34700 1 -- 172.21.15.173:0/4028152972 shutdown_connections 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.924+0000 7f5117a34700 1 --2- 172.21.15.173:0/4028152972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51100f5410 0x7f51100f5830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.924+0000 7f5117a34700 1 -- 172.21.15.173:0/4028152972 >> 172.21.15.173:0/4028152972 conn(0x7f51100f0b70 msgr2=0x7f51100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.924+0000 7f5117a34700 1 -- 172.21.15.173:0/4028152972 shutdown_connections 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.924+0000 7f5117a34700 1 -- 172.21.15.173:0/4028152972 wait complete. 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.924+0000 7f5117a34700 1 Processor -- start 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.925+0000 7f5117a34700 1 -- start start 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.925+0000 7f5117a34700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51100f5410 0x7f51101067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.925+0000 7f5117a34700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5110106ca0 con 0x7f51100f5410 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.925+0000 7f51157d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51100f5410 0x7f51101067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.925+0000 7f51157d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51100f5410 0x7f51101067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:33650/0 (socket says 172.21.15.173:33650) 2024-02-19T22:01:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.925+0000 7f51157d0700 1 -- 172.21.15.173:0/507921487 learned_addr learned my addr 172.21.15.173:0/507921487 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:00.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.925+0000 7f51157d0700 1 -- 172.21.15.173:0/507921487 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c005040 con 0x7f51100f5410 2024-02-19T22:01:00.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.925+0000 7f51157d0700 1 --2- 172.21.15.173:0/507921487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51100f5410 0x7f51101067b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f510c005010 tx=0x7f510c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:00.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.926+0000 7f5105ffb700 1 -- 172.21.15.173:0/507921487 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f510c009320 con 0x7f51100f5410 2024-02-19T22:01:00.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.926+0000 7f5105ffb700 1 -- 172.21.15.173:0/507921487 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f510c013380 con 0x7f51100f5410 2024-02-19T22:01:00.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.926+0000 7f5117a34700 1 -- 172.21.15.173:0/507921487 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5110106ea0 con 0x7f51100f5410 2024-02-19T22:01:00.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.926+0000 7f5105ffb700 1 -- 172.21.15.173:0/507921487 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f510c014e50 con 0x7f51100f5410 2024-02-19T22:01:00.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.926+0000 7f5117a34700 1 -- 172.21.15.173:0/507921487 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5110107330 con 0x7f51100f5410 2024-02-19T22:01:00.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.927+0000 7f5105ffb700 1 -- 172.21.15.173:0/507921487 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f510c014420 con 0x7f51100f5410 2024-02-19T22:01:00.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.927+0000 7f5117a34700 1 -- 172.21.15.173:0/507921487 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5110045ad0 con 0x7f51100f5410 2024-02-19T22:01:00.928 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.927+0000 7f5105ffb700 1 --2- 172.21.15.173:0/507921487 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f50fc02ff10 0x7f50fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:00.928 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.927+0000 7f5105ffb700 1 -- 172.21.15.173:0/507921487 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f510c040f70 con 0x7f51100f5410 2024-02-19T22:01:00.928 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.927+0000 7f5114fcf700 1 --2- 172.21.15.173:0/507921487 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f50fc02ff10 0x7f50fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:00.928 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.927+0000 7f5114fcf700 1 --2- 172.21.15.173:0/507921487 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f50fc02ff10 0x7f50fc0323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f51000096a0 tx=0x7f5100006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:00.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:00.932+0000 7f5105ffb700 1 -- 172.21.15.173:0/507921487 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f510c0095f0 con 0x7f51100f5410 2024-02-19T22:01:01.137 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.137+0000 7f5117a34700 1 -- 172.21.15.173:0/507921487 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51100fafa0 con 0x7f51100f5410 2024-02-19T22:01:01.137 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.138+0000 7f5105ffb700 1 -- 172.21.15.173:0/507921487 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f510c01db70 con 0x7f51100f5410 2024-02-19T22:01:01.138 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:01.138 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:01.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.140+0000 7f5117a34700 1 -- 172.21.15.173:0/507921487 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f50fc02ff10 msgr2=0x7f50fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:01.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.140+0000 7f5117a34700 1 --2- 172.21.15.173:0/507921487 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f50fc02ff10 0x7f50fc0323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f51000096a0 tx=0x7f5100006b40).stop 2024-02-19T22:01:01.139 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.140+0000 7f5117a34700 1 -- 172.21.15.173:0/507921487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51100f5410 msgr2=0x7f51101067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:01.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.140+0000 7f5117a34700 1 --2- 172.21.15.173:0/507921487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51100f5410 0x7f51101067b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f510c005010 tx=0x7f510c00b040).stop 2024-02-19T22:01:01.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.140+0000 7f5117a34700 1 -- 172.21.15.173:0/507921487 shutdown_connections 2024-02-19T22:01:01.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.140+0000 7f5117a34700 1 --2- 172.21.15.173:0/507921487 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f50fc02ff10 0x7f50fc0323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:01.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.140+0000 7f5117a34700 1 --2- 172.21.15.173:0/507921487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51100f5410 0x7f51101067b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:01.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.140+0000 7f5117a34700 1 -- 172.21.15.173:0/507921487 >> 172.21.15.173:0/507921487 conn(0x7f51100f0b70 msgr2=0x7f511005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:01.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.140+0000 7f5117a34700 1 -- 172.21.15.173:0/507921487 shutdown_connections 2024-02-19T22:01:01.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:01.140+0000 7f5117a34700 1 -- 172.21.15.173:0/507921487 wait complete. 2024-02-19T22:01:01.140 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:01.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:01 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:01.139075+0000 mon.smithi143 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.173:0/507921487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:02.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:02.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:00.498481+0000 mgr.smithi143.bissbc (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:04.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.793+0000 7ff1868aa700 1 -- 172.21.15.173:0/3108835610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1800f51a0 msgr2=0x7ff1800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:04.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.793+0000 7ff1868aa700 1 --2- 172.21.15.173:0/3108835610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1800f51a0 0x7ff1800f55c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7ff17c009fb0 tx=0x7ff17c009b30).stop 2024-02-19T22:01:04.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.793+0000 7ff1868aa700 1 -- 172.21.15.173:0/3108835610 shutdown_connections 2024-02-19T22:01:04.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.793+0000 7ff1868aa700 1 --2- 172.21.15.173:0/3108835610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1800f51a0 0x7ff1800f55c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:04.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.793+0000 7ff1868aa700 1 -- 172.21.15.173:0/3108835610 >> 172.21.15.173:0/3108835610 conn(0x7ff1800f09a0 msgr2=0x7ff1800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:04.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.793+0000 7ff1868aa700 1 -- 172.21.15.173:0/3108835610 shutdown_connections 2024-02-19T22:01:04.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.793+0000 7ff1868aa700 1 -- 172.21.15.173:0/3108835610 wait complete. 2024-02-19T22:01:04.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.794+0000 7ff1868aa700 1 Processor -- start 2024-02-19T22:01:04.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.794+0000 7ff1868aa700 1 -- start start 2024-02-19T22:01:04.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.794+0000 7ff1868aa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1800f51a0 0x7ff1801054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:04.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.794+0000 7ff1868aa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1801059c0 con 0x7ff1800f51a0 2024-02-19T22:01:04.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.794+0000 7ff1858a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1800f51a0 0x7ff1801054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:04.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.794+0000 7ff1858a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1800f51a0 0x7ff1801054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:50928/0 (socket says 172.21.15.173:50928) 2024-02-19T22:01:04.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.794+0000 7ff1858a8700 1 -- 172.21.15.173:0/1504527235 learned_addr learned my addr 172.21.15.173:0/1504527235 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:04.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.794+0000 7ff1858a8700 1 -- 172.21.15.173:0/1504527235 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c005040 con 0x7ff1800f51a0 2024-02-19T22:01:04.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.795+0000 7ff1858a8700 1 --2- 172.21.15.173:0/1504527235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1800f51a0 0x7ff1801054d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7ff17c0089f0 tx=0x7ff17c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:04.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.795+0000 7ff1767fc700 1 -- 172.21.15.173:0/1504527235 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff17c009320 con 0x7ff1800f51a0 2024-02-19T22:01:04.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.795+0000 7ff1868aa700 1 -- 172.21.15.173:0/1504527235 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff180105bc0 con 0x7ff1800f51a0 2024-02-19T22:01:04.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.795+0000 7ff1767fc700 1 -- 172.21.15.173:0/1504527235 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff17c013380 con 0x7ff1800f51a0 2024-02-19T22:01:04.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.795+0000 7ff1767fc700 1 -- 172.21.15.173:0/1504527235 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff17c014e20 con 0x7ff1800f51a0 2024-02-19T22:01:04.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.795+0000 7ff1868aa700 1 -- 172.21.15.173:0/1504527235 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff180106050 con 0x7ff1800f51a0 2024-02-19T22:01:04.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.796+0000 7ff1767fc700 1 -- 172.21.15.173:0/1504527235 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff17c0095f0 con 0x7ff1800f51a0 2024-02-19T22:01:04.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.796+0000 7ff1868aa700 1 -- 172.21.15.173:0/1504527235 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff180045ad0 con 0x7ff1800f51a0 2024-02-19T22:01:04.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.796+0000 7ff1767fc700 1 --2- 172.21.15.173:0/1504527235 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff16c032120 0x7ff16c0345e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:04.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.796+0000 7ff1767fc700 1 -- 172.21.15.173:0/1504527235 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff17c040ef0 con 0x7ff1800f51a0 2024-02-19T22:01:04.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.796+0000 7ff1850a7700 1 --2- 172.21.15.173:0/1504527235 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff16c032120 0x7ff16c0345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:04.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.796+0000 7ff1850a7700 1 --2- 172.21.15.173:0/1504527235 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff16c032120 0x7ff16c0345e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7ff1700096a0 tx=0x7ff170006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:04.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:04.801+0000 7ff1767fc700 1 -- 172.21.15.173:0/1504527235 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff17c01d980 con 0x7ff1800f51a0 2024-02-19T22:01:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:02.498771+0000 mgr.smithi143.bissbc (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:05.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.007+0000 7ff1868aa700 1 -- 172.21.15.173:0/1504527235 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff180106f00 con 0x7ff1800f51a0 2024-02-19T22:01:05.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.008+0000 7ff1767fc700 1 -- 172.21.15.173:0/1504527235 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff17c01d980 con 0x7ff1800f51a0 2024-02-19T22:01:05.008 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:05.008 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:05.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.009+0000 7ff1868aa700 1 -- 172.21.15.173:0/1504527235 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff16c032120 msgr2=0x7ff16c0345e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:05.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.009+0000 7ff1868aa700 1 --2- 172.21.15.173:0/1504527235 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff16c032120 0x7ff16c0345e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7ff1700096a0 tx=0x7ff170006b40).stop 2024-02-19T22:01:05.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.009+0000 7ff1868aa700 1 -- 172.21.15.173:0/1504527235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1800f51a0 msgr2=0x7ff1801054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:05.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.009+0000 7ff1868aa700 1 --2- 172.21.15.173:0/1504527235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1800f51a0 0x7ff1801054d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7ff17c0089f0 tx=0x7ff17c00b040).stop 2024-02-19T22:01:05.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.010+0000 7ff1868aa700 1 -- 172.21.15.173:0/1504527235 shutdown_connections 2024-02-19T22:01:05.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.010+0000 7ff1868aa700 1 --2- 172.21.15.173:0/1504527235 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff16c032120 0x7ff16c0345e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:05.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.010+0000 7ff1868aa700 1 --2- 172.21.15.173:0/1504527235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1800f51a0 0x7ff1801054d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:05.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.010+0000 7ff1868aa700 1 -- 172.21.15.173:0/1504527235 >> 172.21.15.173:0/1504527235 conn(0x7ff1800f09a0 msgr2=0x7ff1800f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:05.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.010+0000 7ff1868aa700 1 -- 172.21.15.173:0/1504527235 shutdown_connections 2024-02-19T22:01:05.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:05.010+0000 7ff1868aa700 1 -- 172.21.15.173:0/1504527235 wait complete. 2024-02-19T22:01:05.010 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:05.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:05 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:05.008841+0000 mon.smithi143 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.173:0/1504527235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:06.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:06.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:04.499059+0000 mgr.smithi143.bissbc (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:08.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.807+0000 7ff3ee712700 1 -- 172.21.15.173:0/444370307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e80f5410 msgr2=0x7ff3e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:08.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.807+0000 7ff3ee712700 1 --2- 172.21.15.173:0/444370307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e80f5410 0x7ff3e80f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7ff3d8009fb0 tx=0x7ff3d8009b30).stop 2024-02-19T22:01:08.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.807+0000 7ff3ee712700 1 -- 172.21.15.173:0/444370307 shutdown_connections 2024-02-19T22:01:08.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.807+0000 7ff3ee712700 1 --2- 172.21.15.173:0/444370307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e80f5410 0x7ff3e80f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:08.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.807+0000 7ff3ee712700 1 -- 172.21.15.173:0/444370307 >> 172.21.15.173:0/444370307 conn(0x7ff3e80f0b70 msgr2=0x7ff3e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:08.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.807+0000 7ff3ee712700 1 -- 172.21.15.173:0/444370307 shutdown_connections 2024-02-19T22:01:08.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.807+0000 7ff3ee712700 1 -- 172.21.15.173:0/444370307 wait complete. 2024-02-19T22:01:08.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.808+0000 7ff3ee712700 1 Processor -- start 2024-02-19T22:01:08.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.808+0000 7ff3ee712700 1 -- start start 2024-02-19T22:01:08.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.808+0000 7ff3ee712700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e80f5410 0x7ff3e81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:08.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.808+0000 7ff3ee712700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e8105bd0 con 0x7ff3e80f5410 2024-02-19T22:01:08.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.808+0000 7ff3e7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e80f5410 0x7ff3e81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:08.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.808+0000 7ff3e7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e80f5410 0x7ff3e81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:50948/0 (socket says 172.21.15.173:50948) 2024-02-19T22:01:08.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.808+0000 7ff3e7fff700 1 -- 172.21.15.173:0/417026149 learned_addr learned my addr 172.21.15.173:0/417026149 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:08.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.808+0000 7ff3e7fff700 1 -- 172.21.15.173:0/417026149 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d8005040 con 0x7ff3e80f5410 2024-02-19T22:01:08.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.809+0000 7ff3e7fff700 1 --2- 172.21.15.173:0/417026149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e80f5410 0x7ff3e81056e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7ff3d8005010 tx=0x7ff3d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:08.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.809+0000 7ff3e4ff9700 1 -- 172.21.15.173:0/417026149 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3d8009320 con 0x7ff3e80f5410 2024-02-19T22:01:08.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.809+0000 7ff3ee712700 1 -- 172.21.15.173:0/417026149 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e8105dd0 con 0x7ff3e80f5410 2024-02-19T22:01:08.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.809+0000 7ff3e4ff9700 1 -- 172.21.15.173:0/417026149 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3d8013380 con 0x7ff3e80f5410 2024-02-19T22:01:08.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.809+0000 7ff3e4ff9700 1 -- 172.21.15.173:0/417026149 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3d8014e50 con 0x7ff3e80f5410 2024-02-19T22:01:08.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.809+0000 7ff3ee712700 1 -- 172.21.15.173:0/417026149 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e8106260 con 0x7ff3e80f5410 2024-02-19T22:01:08.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.810+0000 7ff3e4ff9700 1 -- 172.21.15.173:0/417026149 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3d80263a0 con 0x7ff3e80f5410 2024-02-19T22:01:08.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.810+0000 7ff3e4ff9700 1 --2- 172.21.15.173:0/417026149 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d002ff10 0x7ff3d00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:08.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.810+0000 7ff3e4ff9700 1 -- 172.21.15.173:0/417026149 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3d8040f90 con 0x7ff3e80f5410 2024-02-19T22:01:08.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.810+0000 7ff3ee712700 1 -- 172.21.15.173:0/417026149 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e8045ad0 con 0x7ff3e80f5410 2024-02-19T22:01:08.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.810+0000 7ff3e77fe700 1 --2- 172.21.15.173:0/417026149 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d002ff10 0x7ff3d00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:08.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.810+0000 7ff3e77fe700 1 --2- 172.21.15.173:0/417026149 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d002ff10 0x7ff3d00323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff3dc0096a0 tx=0x7ff3dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:08.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:08.815+0000 7ff3e4ff9700 1 -- 172.21.15.173:0/417026149 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3d8012920 con 0x7ff3e80f5410 2024-02-19T22:01:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:06.499386+0000 mgr.smithi143.bissbc (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:09.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.022+0000 7ff3ee712700 1 -- 172.21.15.173:0/417026149 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3e8106fb0 con 0x7ff3e80f5410 2024-02-19T22:01:09.022 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.023+0000 7ff3e4ff9700 1 -- 172.21.15.173:0/417026149 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff3d802b020 con 0x7ff3e80f5410 2024-02-19T22:01:09.022 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:09.022 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:09.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.024+0000 7ff3ee712700 1 -- 172.21.15.173:0/417026149 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d002ff10 msgr2=0x7ff3d00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:09.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.024+0000 7ff3ee712700 1 --2- 172.21.15.173:0/417026149 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d002ff10 0x7ff3d00323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff3dc0096a0 tx=0x7ff3dc006b40).stop 2024-02-19T22:01:09.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.024+0000 7ff3ee712700 1 -- 172.21.15.173:0/417026149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e80f5410 msgr2=0x7ff3e81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:09.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.024+0000 7ff3ee712700 1 --2- 172.21.15.173:0/417026149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e80f5410 0x7ff3e81056e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7ff3d8005010 tx=0x7ff3d800b040).stop 2024-02-19T22:01:09.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.024+0000 7ff3ee712700 1 -- 172.21.15.173:0/417026149 shutdown_connections 2024-02-19T22:01:09.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.024+0000 7ff3ee712700 1 --2- 172.21.15.173:0/417026149 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d002ff10 0x7ff3d00323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:09.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.024+0000 7ff3ee712700 1 --2- 172.21.15.173:0/417026149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3e80f5410 0x7ff3e81056e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:09.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.024+0000 7ff3ee712700 1 -- 172.21.15.173:0/417026149 >> 172.21.15.173:0/417026149 conn(0x7ff3e80f0b70 msgr2=0x7ff3e80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:09.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.025+0000 7ff3ee712700 1 -- 172.21.15.173:0/417026149 shutdown_connections 2024-02-19T22:01:09.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:09.025+0000 7ff3ee712700 1 -- 172.21.15.173:0/417026149 wait complete. 2024-02-19T22:01:09.025 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:09.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:09 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:09.023885+0000 mon.smithi143 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.173:0/417026149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:08.499783+0000 mgr.smithi143.bissbc (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:11.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:11.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:12.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:10.500134+0000 mgr.smithi143.bissbc (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:13.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.124+0000 7f367b59e700 1 -- 172.21.15.173:0/3132783902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f367c0f51a0 msgr2=0x7f367c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:13.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.124+0000 7f367b59e700 1 --2- 172.21.15.173:0/3132783902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f367c0f51a0 0x7f367c0f55c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f366c009fb0 tx=0x7f366c009b30).stop 2024-02-19T22:01:13.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.124+0000 7f367b59e700 1 -- 172.21.15.173:0/3132783902 shutdown_connections 2024-02-19T22:01:13.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.124+0000 7f367b59e700 1 --2- 172.21.15.173:0/3132783902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f367c0f51a0 0x7f367c0f55c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:13.141 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.124+0000 7f367b59e700 1 -- 172.21.15.173:0/3132783902 >> 172.21.15.173:0/3132783902 conn(0x7f367c0f09a0 msgr2=0x7f367c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.124+0000 7f367b59e700 1 -- 172.21.15.173:0/3132783902 shutdown_connections 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.124+0000 7f367b59e700 1 -- 172.21.15.173:0/3132783902 wait complete. 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.124+0000 7f367b59e700 1 Processor -- start 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f367b59e700 1 -- start start 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f367b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f367c0f51a0 0x7f367c0f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f367b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c0f13e0 con 0x7f367c0f51a0 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f367a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f367c0f51a0 0x7f367c0f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f367a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f367c0f51a0 0x7f367c0f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:50182/0 (socket says 172.21.15.173:50182) 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f367a59c700 1 -- 172.21.15.173:0/1322210223 learned_addr learned my addr 172.21.15.173:0/1322210223 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f367a59c700 1 -- 172.21.15.173:0/1322210223 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c005040 con 0x7f367c0f51a0 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f367a59c700 1 --2- 172.21.15.173:0/1322210223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f367c0f51a0 0x7f367c0f2f60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f366c0089f0 tx=0x7f366c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f366affd700 1 -- 172.21.15.173:0/1322210223 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f366c009320 con 0x7f367c0f51a0 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f367b59e700 1 -- 172.21.15.173:0/1322210223 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367c0f1580 con 0x7f367c0f51a0 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f366affd700 1 -- 172.21.15.173:0/1322210223 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f366c013380 con 0x7f367c0f51a0 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f366affd700 1 -- 172.21.15.173:0/1322210223 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f366c014e20 con 0x7f367c0f51a0 2024-02-19T22:01:13.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.125+0000 7f367b59e700 1 -- 172.21.15.173:0/1322210223 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367c0f1a10 con 0x7f367c0f51a0 2024-02-19T22:01:13.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.126+0000 7f366affd700 1 -- 172.21.15.173:0/1322210223 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f366c0095f0 con 0x7f367c0f51a0 2024-02-19T22:01:13.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.126+0000 7f367b59e700 1 -- 172.21.15.173:0/1322210223 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367c059100 con 0x7f367c0f51a0 2024-02-19T22:01:13.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.126+0000 7f366affd700 1 --2- 172.21.15.173:0/1322210223 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f366402ff10 0x7f36640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:13.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.127+0000 7f366affd700 1 -- 172.21.15.173:0/1322210223 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f366c040e90 con 0x7f367c0f51a0 2024-02-19T22:01:13.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.127+0000 7f3679d9b700 1 --2- 172.21.15.173:0/1322210223 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f366402ff10 0x7f36640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:13.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.127+0000 7f3679d9b700 1 --2- 172.21.15.173:0/1322210223 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f366402ff10 0x7f36640323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f36700096a0 tx=0x7f3670006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:13.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.132+0000 7f366affd700 1 -- 172.21.15.173:0/1322210223 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f366c014420 con 0x7f367c0f51a0 2024-02-19T22:01:13.339 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.339+0000 7f367b59e700 1 -- 172.21.15.173:0/1322210223 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f367c045ad0 con 0x7f367c0f51a0 2024-02-19T22:01:13.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.340+0000 7f366affd700 1 -- 172.21.15.173:0/1322210223 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f366c01d980 con 0x7f367c0f51a0 2024-02-19T22:01:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:13.340 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:13.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.342+0000 7f367b59e700 1 -- 172.21.15.173:0/1322210223 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f366402ff10 msgr2=0x7f36640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:13.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.342+0000 7f367b59e700 1 --2- 172.21.15.173:0/1322210223 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f366402ff10 0x7f36640323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f36700096a0 tx=0x7f3670006b40).stop 2024-02-19T22:01:13.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.342+0000 7f367b59e700 1 -- 172.21.15.173:0/1322210223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f367c0f51a0 msgr2=0x7f367c0f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:13.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.342+0000 7f367b59e700 1 --2- 172.21.15.173:0/1322210223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f367c0f51a0 0x7f367c0f2f60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f366c0089f0 tx=0x7f366c00b040).stop 2024-02-19T22:01:13.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.342+0000 7f367b59e700 1 -- 172.21.15.173:0/1322210223 shutdown_connections 2024-02-19T22:01:13.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.342+0000 7f367b59e700 1 --2- 172.21.15.173:0/1322210223 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f366402ff10 0x7f36640323d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:13.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.342+0000 7f367b59e700 1 --2- 172.21.15.173:0/1322210223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f367c0f51a0 0x7f367c0f2f60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:13.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.342+0000 7f367b59e700 1 -- 172.21.15.173:0/1322210223 >> 172.21.15.173:0/1322210223 conn(0x7f367c0f09a0 msgr2=0x7f367c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:13.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.342+0000 7f367b59e700 1 -- 172.21.15.173:0/1322210223 shutdown_connections 2024-02-19T22:01:13.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:13.342+0000 7f367b59e700 1 -- 172.21.15.173:0/1322210223 wait complete. 2024-02-19T22:01:13.343 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:13.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:13 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:13.341439+0000 mon.smithi143 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.173:0/1322210223' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:12.500465+0000 mgr.smithi143.bissbc (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:15.197 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:15.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:14.500755+0000 mgr.smithi143.bissbc (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:17.213 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.186+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/4270294355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c080f5410 msgr2=0x7f0c080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:17.213 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.186+0000 7f0c0f9d9700 1 --2- 172.21.15.173:0/4270294355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c080f5410 0x7f0c080f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f0bf8009fb0 tx=0x7f0bf8009b30).stop 2024-02-19T22:01:17.213 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.186+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/4270294355 shutdown_connections 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.186+0000 7f0c0f9d9700 1 --2- 172.21.15.173:0/4270294355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c080f5410 0x7f0c080f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.186+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/4270294355 >> 172.21.15.173:0/4270294355 conn(0x7f0c080f0b70 msgr2=0x7f0c080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.186+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/4270294355 shutdown_connections 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.186+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/4270294355 wait complete. 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.187+0000 7f0c0f9d9700 1 Processor -- start 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.187+0000 7f0c0f9d9700 1 -- start start 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.187+0000 7f0c0f9d9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c080f5410 0x7f0c081094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.187+0000 7f0c0f9d9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c08060240 con 0x7f0c080f5410 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.187+0000 7f0c0d775700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c080f5410 0x7f0c081094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.187+0000 7f0c0d775700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c080f5410 0x7f0c081094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:50210/0 (socket says 172.21.15.173:50210) 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.187+0000 7f0c0d775700 1 -- 172.21.15.173:0/3577166712 learned_addr learned my addr 172.21.15.173:0/3577166712 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.187+0000 7f0c0d775700 1 -- 172.21.15.173:0/3577166712 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf8005040 con 0x7f0c080f5410 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.187+0000 7f0c0d775700 1 --2- 172.21.15.173:0/3577166712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c080f5410 0x7f0c081094e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f0bf8005010 tx=0x7f0bf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:17.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.188+0000 7f0bfdffb700 1 -- 172.21.15.173:0/3577166712 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bf8009320 con 0x7f0c080f5410 2024-02-19T22:01:17.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.188+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/3577166712 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c08060440 con 0x7f0c080f5410 2024-02-19T22:01:17.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.188+0000 7f0bfdffb700 1 -- 172.21.15.173:0/3577166712 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bf8013380 con 0x7f0c080f5410 2024-02-19T22:01:17.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.188+0000 7f0bfdffb700 1 -- 172.21.15.173:0/3577166712 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bf8014e50 con 0x7f0c080f5410 2024-02-19T22:01:17.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.188+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/3577166712 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c08109b90 con 0x7f0c080f5410 2024-02-19T22:01:17.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.188+0000 7f0bfdffb700 1 -- 172.21.15.173:0/3577166712 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0bf8014420 con 0x7f0c080f5410 2024-02-19T22:01:17.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.189+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/3577166712 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c08045ad0 con 0x7f0c080f5410 2024-02-19T22:01:17.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.189+0000 7f0bfdffb700 1 --2- 172.21.15.173:0/3577166712 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0bf402ff10 0x7f0bf40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:17.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.189+0000 7f0bfdffb700 1 -- 172.21.15.173:0/3577166712 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0bf8040e60 con 0x7f0c080f5410 2024-02-19T22:01:17.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.189+0000 7f0c0cf74700 1 --2- 172.21.15.173:0/3577166712 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0bf402ff10 0x7f0bf40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:17.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.189+0000 7f0c0cf74700 1 --2- 172.21.15.173:0/3577166712 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0bf402ff10 0x7f0bf40323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f0c040096a0 tx=0x7f0c04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:17.215 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.194+0000 7f0bfdffb700 1 -- 172.21.15.173:0/3577166712 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0bf80095f0 con 0x7f0c080f5410 2024-02-19T22:01:17.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.399+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/3577166712 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c0810a3b0 con 0x7f0c080f5410 2024-02-19T22:01:17.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.400+0000 7f0bfdffb700 1 -- 172.21.15.173:0/3577166712 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0bf801db70 con 0x7f0c080f5410 2024-02-19T22:01:17.400 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:17.400 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:17.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.403+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/3577166712 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0bf402ff10 msgr2=0x7f0bf40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:17.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.403+0000 7f0c0f9d9700 1 --2- 172.21.15.173:0/3577166712 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0bf402ff10 0x7f0bf40323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f0c040096a0 tx=0x7f0c04006b40).stop 2024-02-19T22:01:17.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.403+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/3577166712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c080f5410 msgr2=0x7f0c081094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:17.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.403+0000 7f0c0f9d9700 1 --2- 172.21.15.173:0/3577166712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c080f5410 0x7f0c081094e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f0bf8005010 tx=0x7f0bf800b040).stop 2024-02-19T22:01:17.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.403+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/3577166712 shutdown_connections 2024-02-19T22:01:17.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.403+0000 7f0c0f9d9700 1 --2- 172.21.15.173:0/3577166712 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f0bf402ff10 0x7f0bf40323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:17.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.403+0000 7f0c0f9d9700 1 --2- 172.21.15.173:0/3577166712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c080f5410 0x7f0c081094e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:17.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.403+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/3577166712 >> 172.21.15.173:0/3577166712 conn(0x7f0c080f0b70 msgr2=0x7f0c080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:17.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.403+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/3577166712 shutdown_connections 2024-02-19T22:01:17.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:17.403+0000 7f0c0f9d9700 1 -- 172.21.15.173:0/3577166712 wait complete. 2024-02-19T22:01:17.404 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:17.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:17 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:17.401399+0000 mon.smithi143 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.173:0/3577166712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:16.500969+0000 mgr.smithi143.bissbc (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:19.160 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:19.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:19.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:19 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:18.501298+0000 mgr.smithi143.bissbc (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:21.340 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.339+0000 7f2cf07d0700 1 -- 172.21.15.173:0/460057680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80f4740 msgr2=0x7f2ce80f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.339+0000 7f2cf07d0700 1 --2- 172.21.15.173:0/460057680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80f4740 0x7f2ce80f4b60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f2cd8009fb0 tx=0x7f2cd8009b30).stop 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.339+0000 7f2cf07d0700 1 -- 172.21.15.173:0/460057680 shutdown_connections 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.339+0000 7f2cf07d0700 1 --2- 172.21.15.173:0/460057680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80f4740 0x7f2ce80f4b60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.339+0000 7f2cf07d0700 1 -- 172.21.15.173:0/460057680 >> 172.21.15.173:0/460057680 conn(0x7f2ce80f0260 msgr2=0x7f2ce80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.339+0000 7f2cf07d0700 1 -- 172.21.15.173:0/460057680 shutdown_connections 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.339+0000 7f2cf07d0700 1 -- 172.21.15.173:0/460057680 wait complete. 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.340+0000 7f2cf07d0700 1 Processor -- start 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.340+0000 7f2cf07d0700 1 -- start start 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.340+0000 7f2cf07d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80f4740 0x7f2ce81067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.340+0000 7f2cf07d0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce8106ca0 con 0x7f2ce80f4740 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.340+0000 7f2cee56c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80f4740 0x7f2ce81067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:21.341 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.340+0000 7f2cee56c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80f4740 0x7f2ce81067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43868/0 (socket says 172.21.15.173:43868) 2024-02-19T22:01:21.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.340+0000 7f2cee56c700 1 -- 172.21.15.173:0/2487467472 learned_addr learned my addr 172.21.15.173:0/2487467472 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:21.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.340+0000 7f2cee56c700 1 -- 172.21.15.173:0/2487467472 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd8005040 con 0x7f2ce80f4740 2024-02-19T22:01:21.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.341+0000 7f2cee56c700 1 --2- 172.21.15.173:0/2487467472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80f4740 0x7f2ce81067b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f2cd8005010 tx=0x7f2cd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:21.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.341+0000 7f2cdeffd700 1 -- 172.21.15.173:0/2487467472 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cd8009320 con 0x7f2ce80f4740 2024-02-19T22:01:21.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.341+0000 7f2cdeffd700 1 -- 172.21.15.173:0/2487467472 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cd8013380 con 0x7f2ce80f4740 2024-02-19T22:01:21.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.341+0000 7f2cf07d0700 1 -- 172.21.15.173:0/2487467472 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce8106ea0 con 0x7f2ce80f4740 2024-02-19T22:01:21.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.341+0000 7f2cdeffd700 1 -- 172.21.15.173:0/2487467472 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cd8014e50 con 0x7f2ce80f4740 2024-02-19T22:01:21.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.341+0000 7f2cf07d0700 1 -- 172.21.15.173:0/2487467472 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce8107330 con 0x7f2ce80f4740 2024-02-19T22:01:21.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.342+0000 7f2cdeffd700 1 -- 172.21.15.173:0/2487467472 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2cd8014420 con 0x7f2ce80f4740 2024-02-19T22:01:21.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.342+0000 7f2cf07d0700 1 -- 172.21.15.173:0/2487467472 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce8045ad0 con 0x7f2ce80f4740 2024-02-19T22:01:21.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.342+0000 7f2cdeffd700 1 --2- 172.21.15.173:0/2487467472 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2cd402ff10 0x7f2cd40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:21.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.342+0000 7f2cdeffd700 1 -- 172.21.15.173:0/2487467472 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2cd8040e60 con 0x7f2ce80f4740 2024-02-19T22:01:21.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.342+0000 7f2cedd6b700 1 --2- 172.21.15.173:0/2487467472 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2cd402ff10 0x7f2cd40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:21.343 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.343+0000 7f2cedd6b700 1 --2- 172.21.15.173:0/2487467472 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2cd402ff10 0x7f2cd40323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f2ce40096a0 tx=0x7f2ce4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:21.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.347+0000 7f2cdeffd700 1 -- 172.21.15.173:0/2487467472 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2cd80095f0 con 0x7f2ce80f4740 2024-02-19T22:01:21.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.554+0000 7f2cf07d0700 1 -- 172.21.15.173:0/2487467472 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ce81081e0 con 0x7f2ce80f4740 2024-02-19T22:01:21.554 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.556+0000 7f2cdeffd700 1 -- 172.21.15.173:0/2487467472 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2cd801db70 con 0x7f2ce80f4740 2024-02-19T22:01:21.555 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:21.555 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:21.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.557+0000 7f2cf07d0700 1 -- 172.21.15.173:0/2487467472 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2cd402ff10 msgr2=0x7f2cd40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:21.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.557+0000 7f2cf07d0700 1 --2- 172.21.15.173:0/2487467472 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2cd402ff10 0x7f2cd40323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f2ce40096a0 tx=0x7f2ce4006b40).stop 2024-02-19T22:01:21.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.557+0000 7f2cf07d0700 1 -- 172.21.15.173:0/2487467472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80f4740 msgr2=0x7f2ce81067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:21.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.557+0000 7f2cf07d0700 1 --2- 172.21.15.173:0/2487467472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80f4740 0x7f2ce81067b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f2cd8005010 tx=0x7f2cd800b040).stop 2024-02-19T22:01:21.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.557+0000 7f2cf07d0700 1 -- 172.21.15.173:0/2487467472 shutdown_connections 2024-02-19T22:01:21.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.557+0000 7f2cf07d0700 1 --2- 172.21.15.173:0/2487467472 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2cd402ff10 0x7f2cd40323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:21.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.557+0000 7f2cf07d0700 1 --2- 172.21.15.173:0/2487467472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80f4740 0x7f2ce81067b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:21.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.557+0000 7f2cf07d0700 1 -- 172.21.15.173:0/2487467472 >> 172.21.15.173:0/2487467472 conn(0x7f2ce80f0260 msgr2=0x7f2ce80f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:21.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.557+0000 7f2cf07d0700 1 -- 172.21.15.173:0/2487467472 shutdown_connections 2024-02-19T22:01:21.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:21.557+0000 7f2cf07d0700 1 -- 172.21.15.173:0/2487467472 wait complete. 2024-02-19T22:01:21.558 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:20.501557+0000 mgr.smithi143.bissbc (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:20.568782+0000 mon.smithi143 (mon.0) 237 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T22:01:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:20.568982+0000 mon.smithi143 (mon.0) 238 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:01:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:21.556096+0000 mon.smithi143 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.173:0/2487467472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:23.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:23.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:24.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:22.501884+0000 mgr.smithi143.bissbc (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:25.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.456+0000 7f189b2fe700 1 -- 172.21.15.173:0/552584150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18940f5410 msgr2=0x7f18940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:25.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.456+0000 7f189b2fe700 1 --2- 172.21.15.173:0/552584150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18940f5410 0x7f18940f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f1884009fb0 tx=0x7f1884009b30).stop 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.456+0000 7f189b2fe700 1 -- 172.21.15.173:0/552584150 shutdown_connections 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.456+0000 7f189b2fe700 1 --2- 172.21.15.173:0/552584150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18940f5410 0x7f18940f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.456+0000 7f189b2fe700 1 -- 172.21.15.173:0/552584150 >> 172.21.15.173:0/552584150 conn(0x7f18940f0b70 msgr2=0x7f18940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.456+0000 7f189b2fe700 1 -- 172.21.15.173:0/552584150 shutdown_connections 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.456+0000 7f189b2fe700 1 -- 172.21.15.173:0/552584150 wait complete. 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.456+0000 7f189b2fe700 1 Processor -- start 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.457+0000 7f189b2fe700 1 -- start start 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.457+0000 7f189b2fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18940fd520 0x7f18940fb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.457+0000 7f189b2fe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18940fd940 con 0x7f18940fd520 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.457+0000 7f189909a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18940fd520 0x7f18940fb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.457+0000 7f189909a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18940fd520 0x7f18940fb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43884/0 (socket says 172.21.15.173:43884) 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.457+0000 7f189909a700 1 -- 172.21.15.173:0/3546307306 learned_addr learned my addr 172.21.15.173:0/3546307306 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:25.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.457+0000 7f189909a700 1 -- 172.21.15.173:0/3546307306 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1884005040 con 0x7f18940fd520 2024-02-19T22:01:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.457+0000 7f189909a700 1 --2- 172.21.15.173:0/3546307306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18940fd520 0x7f18940fb9e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f1884009f80 tx=0x7f1884009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.457+0000 7f1889ffb700 1 -- 172.21.15.173:0/3546307306 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1884013380 con 0x7f18940fd520 2024-02-19T22:01:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.458+0000 7f189b2fe700 1 -- 172.21.15.173:0/3546307306 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18940fbed0 con 0x7f18940fd520 2024-02-19T22:01:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.458+0000 7f1889ffb700 1 -- 172.21.15.173:0/3546307306 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1884013790 con 0x7f18940fd520 2024-02-19T22:01:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.458+0000 7f1889ffb700 1 -- 172.21.15.173:0/3546307306 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1884014420 con 0x7f18940fd520 2024-02-19T22:01:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.458+0000 7f189b2fe700 1 -- 172.21.15.173:0/3546307306 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18940fc3c0 con 0x7f18940fd520 2024-02-19T22:01:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.458+0000 7f1889ffb700 1 -- 172.21.15.173:0/3546307306 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1884025060 con 0x7f18940fd520 2024-02-19T22:01:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.459+0000 7f189b2fe700 1 -- 172.21.15.173:0/3546307306 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1894059100 con 0x7f18940fd520 2024-02-19T22:01:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.459+0000 7f1889ffb700 1 --2- 172.21.15.173:0/3546307306 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f188002ff60 0x7f1880032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.459+0000 7f1889ffb700 1 -- 172.21.15.173:0/3546307306 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1884040d80 con 0x7f18940fd520 2024-02-19T22:01:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.459+0000 7f1898899700 1 --2- 172.21.15.173:0/3546307306 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f188002ff60 0x7f1880032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:25.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.459+0000 7f1898899700 1 --2- 172.21.15.173:0/3546307306 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f188002ff60 0x7f1880032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f18900096a0 tx=0x7f1890006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:25.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.464+0000 7f1889ffb700 1 -- 172.21.15.173:0/3546307306 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f188401c950 con 0x7f18940fd520 2024-02-19T22:01:25.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.670+0000 7f189b2fe700 1 -- 172.21.15.173:0/3546307306 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1894045ad0 con 0x7f18940fd520 2024-02-19T22:01:25.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.671+0000 7f1889ffb700 1 -- 172.21.15.173:0/3546307306 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f188402d2d0 con 0x7f18940fd520 2024-02-19T22:01:25.671 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:25.671 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:25.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.673+0000 7f189b2fe700 1 -- 172.21.15.173:0/3546307306 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f188002ff60 msgr2=0x7f1880032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:25.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.673+0000 7f189b2fe700 1 --2- 172.21.15.173:0/3546307306 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f188002ff60 0x7f1880032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f18900096a0 tx=0x7f1890006b40).stop 2024-02-19T22:01:25.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.673+0000 7f189b2fe700 1 -- 172.21.15.173:0/3546307306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18940fd520 msgr2=0x7f18940fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:25.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.673+0000 7f189b2fe700 1 --2- 172.21.15.173:0/3546307306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18940fd520 0x7f18940fb9e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f1884009f80 tx=0x7f1884009b30).stop 2024-02-19T22:01:25.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.673+0000 7f189b2fe700 1 -- 172.21.15.173:0/3546307306 shutdown_connections 2024-02-19T22:01:25.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.673+0000 7f189b2fe700 1 --2- 172.21.15.173:0/3546307306 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f188002ff60 0x7f1880032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:25.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.673+0000 7f189b2fe700 1 --2- 172.21.15.173:0/3546307306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18940fd520 0x7f18940fb9e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:25.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.673+0000 7f189b2fe700 1 -- 172.21.15.173:0/3546307306 >> 172.21.15.173:0/3546307306 conn(0x7f18940f0b70 msgr2=0x7f189405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:25.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.673+0000 7f189b2fe700 1 -- 172.21.15.173:0/3546307306 shutdown_connections 2024-02-19T22:01:25.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:25.673+0000 7f189b2fe700 1 -- 172.21.15.173:0/3546307306 wait complete. 2024-02-19T22:01:25.673 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:24.502137+0000 mgr.smithi143.bissbc (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:25.672006+0000 mon.smithi143 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.173:0/3546307306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:27.423 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:27.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:26.502485+0000 mgr.smithi143.bissbc (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:29.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.410+0000 7fed39ed2700 1 -- 172.21.15.173:0/1093132189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34060180 msgr2=0x7fed340f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:29.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.410+0000 7fed39ed2700 1 --2- 172.21.15.173:0/1093132189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34060180 0x7fed340f8f80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fed28009fb0 tx=0x7fed28009b30).stop 2024-02-19T22:01:29.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.410+0000 7fed39ed2700 1 -- 172.21.15.173:0/1093132189 shutdown_connections 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.410+0000 7fed39ed2700 1 --2- 172.21.15.173:0/1093132189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34060180 0x7fed340f8f80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.410+0000 7fed39ed2700 1 -- 172.21.15.173:0/1093132189 >> 172.21.15.173:0/1093132189 conn(0x7fed340f09a0 msgr2=0x7fed340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.410+0000 7fed39ed2700 1 -- 172.21.15.173:0/1093132189 shutdown_connections 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.410+0000 7fed39ed2700 1 -- 172.21.15.173:0/1093132189 wait complete. 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.410+0000 7fed39ed2700 1 Processor -- start 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.411+0000 7fed39ed2700 1 -- start start 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.411+0000 7fed39ed2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34060180 0x7fed340f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.411+0000 7fed39ed2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed340f4e80 con 0x7fed34060180 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.411+0000 7fed38ed0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34060180 0x7fed340f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.411+0000 7fed38ed0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34060180 0x7fed340f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43906/0 (socket says 172.21.15.173:43906) 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.411+0000 7fed38ed0700 1 -- 172.21.15.173:0/724722109 learned_addr learned my addr 172.21.15.173:0/724722109 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:29.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.411+0000 7fed38ed0700 1 -- 172.21.15.173:0/724722109 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed28005040 con 0x7fed34060180 2024-02-19T22:01:29.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.412+0000 7fed38ed0700 1 --2- 172.21.15.173:0/724722109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34060180 0x7fed340f6a00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fed28005010 tx=0x7fed2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:29.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.412+0000 7fed317fa700 1 -- 172.21.15.173:0/724722109 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed28009320 con 0x7fed34060180 2024-02-19T22:01:29.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.412+0000 7fed39ed2700 1 -- 172.21.15.173:0/724722109 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed340f5020 con 0x7fed34060180 2024-02-19T22:01:29.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.412+0000 7fed317fa700 1 -- 172.21.15.173:0/724722109 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed28013380 con 0x7fed34060180 2024-02-19T22:01:29.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.412+0000 7fed317fa700 1 -- 172.21.15.173:0/724722109 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed28014e50 con 0x7fed34060180 2024-02-19T22:01:29.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.412+0000 7fed39ed2700 1 -- 172.21.15.173:0/724722109 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed340f54b0 con 0x7fed34060180 2024-02-19T22:01:29.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.412+0000 7fed317fa700 1 -- 172.21.15.173:0/724722109 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed280263a0 con 0x7fed34060180 2024-02-19T22:01:29.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.413+0000 7fed317fa700 1 --2- 172.21.15.173:0/724722109 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fed2402ff10 0x7fed240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:29.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.413+0000 7fed39ed2700 1 -- 172.21.15.173:0/724722109 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed34059100 con 0x7fed34060180 2024-02-19T22:01:29.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.413+0000 7fed317fa700 1 -- 172.21.15.173:0/724722109 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed28040eb0 con 0x7fed34060180 2024-02-19T22:01:29.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.413+0000 7fed33fff700 1 --2- 172.21.15.173:0/724722109 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fed2402ff10 0x7fed240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:29.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.413+0000 7fed33fff700 1 --2- 172.21.15.173:0/724722109 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fed2402ff10 0x7fed240323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fed200096a0 tx=0x7fed20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:29.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.418+0000 7fed317fa700 1 -- 172.21.15.173:0/724722109 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed28012920 con 0x7fed34060180 2024-02-19T22:01:29.622 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.623+0000 7fed39ed2700 1 -- 172.21.15.173:0/724722109 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed340fbe20 con 0x7fed34060180 2024-02-19T22:01:29.623 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.623+0000 7fed317fa700 1 -- 172.21.15.173:0/724722109 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fed28012330 con 0x7fed34060180 2024-02-19T22:01:29.623 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:29.623 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:29.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.625+0000 7fed39ed2700 1 -- 172.21.15.173:0/724722109 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fed2402ff10 msgr2=0x7fed240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:29.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.625+0000 7fed39ed2700 1 --2- 172.21.15.173:0/724722109 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fed2402ff10 0x7fed240323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fed200096a0 tx=0x7fed20006b40).stop 2024-02-19T22:01:29.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.625+0000 7fed39ed2700 1 -- 172.21.15.173:0/724722109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34060180 msgr2=0x7fed340f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:29.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.625+0000 7fed39ed2700 1 --2- 172.21.15.173:0/724722109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34060180 0x7fed340f6a00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fed28005010 tx=0x7fed2800b040).stop 2024-02-19T22:01:29.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.625+0000 7fed39ed2700 1 -- 172.21.15.173:0/724722109 shutdown_connections 2024-02-19T22:01:29.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.625+0000 7fed39ed2700 1 --2- 172.21.15.173:0/724722109 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fed2402ff10 0x7fed240323d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:29.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.625+0000 7fed39ed2700 1 --2- 172.21.15.173:0/724722109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34060180 0x7fed340f6a00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:29.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.625+0000 7fed39ed2700 1 -- 172.21.15.173:0/724722109 >> 172.21.15.173:0/724722109 conn(0x7fed340f09a0 msgr2=0x7fed340f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:29.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.625+0000 7fed39ed2700 1 -- 172.21.15.173:0/724722109 shutdown_connections 2024-02-19T22:01:29.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:29.625+0000 7fed39ed2700 1 -- 172.21.15.173:0/724722109 wait complete. 2024-02-19T22:01:29.626 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:28.502969+0000 mgr.smithi143.bissbc (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:29.624583+0000 mon.smithi143 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.173:0/724722109' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:31.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:31.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:32.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:30.503257+0000 mgr.smithi143.bissbc (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:33.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.624+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/2409389148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db40f5410 msgr2=0x7f6db40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.624+0000 7f6dbb2aa700 1 --2- 172.21.15.173:0/2409389148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db40f5410 0x7f6db40f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f6da4009fb0 tx=0x7f6da4009b30).stop 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.624+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/2409389148 shutdown_connections 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.624+0000 7f6dbb2aa700 1 --2- 172.21.15.173:0/2409389148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db40f5410 0x7f6db40f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.624+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/2409389148 >> 172.21.15.173:0/2409389148 conn(0x7f6db40f0b70 msgr2=0x7f6db40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.624+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/2409389148 shutdown_connections 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.624+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/2409389148 wait complete. 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.625+0000 7f6dbb2aa700 1 Processor -- start 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.625+0000 7f6dbb2aa700 1 -- start start 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.625+0000 7f6dbb2aa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db40f5410 0x7f6db41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.625+0000 7f6dbb2aa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db4105bd0 con 0x7f6db40f5410 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.625+0000 7f6db9046700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db40f5410 0x7f6db41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.625+0000 7f6db9046700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db40f5410 0x7f6db41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:38006/0 (socket says 172.21.15.173:38006) 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.625+0000 7f6db9046700 1 -- 172.21.15.173:0/4201650441 learned_addr learned my addr 172.21.15.173:0/4201650441 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.625+0000 7f6db9046700 1 -- 172.21.15.173:0/4201650441 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da4005040 con 0x7f6db40f5410 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.626+0000 7f6db9046700 1 --2- 172.21.15.173:0/4201650441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db40f5410 0x7f6db41056e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f6da4005010 tx=0x7f6da400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:33.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.626+0000 7f6da9ffb700 1 -- 172.21.15.173:0/4201650441 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6da4009320 con 0x7f6db40f5410 2024-02-19T22:01:33.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.626+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/4201650441 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db4105dd0 con 0x7f6db40f5410 2024-02-19T22:01:33.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.626+0000 7f6da9ffb700 1 -- 172.21.15.173:0/4201650441 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6da4013380 con 0x7f6db40f5410 2024-02-19T22:01:33.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.626+0000 7f6da9ffb700 1 -- 172.21.15.173:0/4201650441 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6da4014e50 con 0x7f6db40f5410 2024-02-19T22:01:33.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.626+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/4201650441 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db4106260 con 0x7f6db40f5410 2024-02-19T22:01:33.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.627+0000 7f6da9ffb700 1 -- 172.21.15.173:0/4201650441 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6da40095f0 con 0x7f6db40f5410 2024-02-19T22:01:33.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.627+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/4201650441 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db4045ad0 con 0x7f6db40f5410 2024-02-19T22:01:33.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.627+0000 7f6da9ffb700 1 --2- 172.21.15.173:0/4201650441 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6da002ff10 0x7f6da00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:33.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.627+0000 7f6da9ffb700 1 -- 172.21.15.173:0/4201650441 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6da4040f80 con 0x7f6db40f5410 2024-02-19T22:01:33.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.627+0000 7f6db8845700 1 --2- 172.21.15.173:0/4201650441 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6da002ff10 0x7f6da00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:33.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.628+0000 7f6db8845700 1 --2- 172.21.15.173:0/4201650441 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6da002ff10 0x7f6da00323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f6db00096a0 tx=0x7f6db0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:33.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.632+0000 7f6da9ffb700 1 -- 172.21.15.173:0/4201650441 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6da4014420 con 0x7f6db40f5410 2024-02-19T22:01:33.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.837+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/4201650441 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6db41070b0 con 0x7f6db40f5410 2024-02-19T22:01:33.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.838+0000 7f6da9ffb700 1 -- 172.21.15.173:0/4201650441 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6da4010230 con 0x7f6db40f5410 2024-02-19T22:01:33.838 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:33.838 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.839+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/4201650441 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6da002ff10 msgr2=0x7f6da00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.839+0000 7f6dbb2aa700 1 --2- 172.21.15.173:0/4201650441 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6da002ff10 0x7f6da00323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f6db00096a0 tx=0x7f6db0006b40).stop 2024-02-19T22:01:33.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.839+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/4201650441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db40f5410 msgr2=0x7f6db41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:33.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.839+0000 7f6dbb2aa700 1 --2- 172.21.15.173:0/4201650441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db40f5410 0x7f6db41056e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f6da4005010 tx=0x7f6da400b040).stop 2024-02-19T22:01:33.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.839+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/4201650441 shutdown_connections 2024-02-19T22:01:33.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.839+0000 7f6dbb2aa700 1 --2- 172.21.15.173:0/4201650441 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6da002ff10 0x7f6da00323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:33.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.839+0000 7f6dbb2aa700 1 --2- 172.21.15.173:0/4201650441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db40f5410 0x7f6db41056e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:33.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.839+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/4201650441 >> 172.21.15.173:0/4201650441 conn(0x7f6db40f0b70 msgr2=0x7f6db40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:33.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.840+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/4201650441 shutdown_connections 2024-02-19T22:01:33.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:33.840+0000 7f6dbb2aa700 1 -- 172.21.15.173:0/4201650441 wait complete. 2024-02-19T22:01:33.841 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:32.503529+0000 mgr.smithi143.bissbc (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:33.839773+0000 mon.smithi143 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.173:0/4201650441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:35.610 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:35.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:34.503962+0000 mgr.smithi143.bissbc (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:37.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.541+0000 7f14f07aa700 1 -- 172.21.15.173:0/55184312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e80f6510 msgr2=0x7f14e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:37.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.541+0000 7f14f07aa700 1 --2- 172.21.15.173:0/55184312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e80f6510 0x7f14e80f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f14d8009fb0 tx=0x7f14d8009b30).stop 2024-02-19T22:01:37.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.541+0000 7f14f07aa700 1 -- 172.21.15.173:0/55184312 shutdown_connections 2024-02-19T22:01:37.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.541+0000 7f14f07aa700 1 --2- 172.21.15.173:0/55184312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e80f6510 0x7f14e80f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:37.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.541+0000 7f14f07aa700 1 -- 172.21.15.173:0/55184312 >> 172.21.15.173:0/55184312 conn(0x7f14e80f1c70 msgr2=0x7f14e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:37.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.541+0000 7f14f07aa700 1 -- 172.21.15.173:0/55184312 shutdown_connections 2024-02-19T22:01:37.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.541+0000 7f14f07aa700 1 -- 172.21.15.173:0/55184312 wait complete. 2024-02-19T22:01:37.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.541+0000 7f14f07aa700 1 Processor -- start 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.541+0000 7f14f07aa700 1 -- start start 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.541+0000 7f14f07aa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e80f6510 0x7f14e81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.541+0000 7f14f07aa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e8106cd0 con 0x7f14e80f6510 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.542+0000 7f14ee546700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e80f6510 0x7f14e81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.542+0000 7f14ee546700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e80f6510 0x7f14e81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:38032/0 (socket says 172.21.15.173:38032) 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.542+0000 7f14ee546700 1 -- 172.21.15.173:0/550030264 learned_addr learned my addr 172.21.15.173:0/550030264 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.542+0000 7f14ee546700 1 -- 172.21.15.173:0/550030264 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8005040 con 0x7f14e80f6510 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.542+0000 7f14ee546700 1 --2- 172.21.15.173:0/550030264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e80f6510 0x7f14e81067e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f14d8005010 tx=0x7f14d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.542+0000 7f14deffd700 1 -- 172.21.15.173:0/550030264 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14d8009320 con 0x7f14e80f6510 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.542+0000 7f14f07aa700 1 -- 172.21.15.173:0/550030264 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e8106ed0 con 0x7f14e80f6510 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.542+0000 7f14deffd700 1 -- 172.21.15.173:0/550030264 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14d8013380 con 0x7f14e80f6510 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.542+0000 7f14deffd700 1 -- 172.21.15.173:0/550030264 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14d8014e50 con 0x7f14e80f6510 2024-02-19T22:01:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.542+0000 7f14f07aa700 1 -- 172.21.15.173:0/550030264 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e8107360 con 0x7f14e80f6510 2024-02-19T22:01:37.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.543+0000 7f14deffd700 1 -- 172.21.15.173:0/550030264 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14d80095f0 con 0x7f14e80f6510 2024-02-19T22:01:37.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.543+0000 7f14deffd700 1 --2- 172.21.15.173:0/550030264 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14d402ff10 0x7f14d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:37.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.543+0000 7f14f07aa700 1 -- 172.21.15.173:0/550030264 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e8045ad0 con 0x7f14e80f6510 2024-02-19T22:01:37.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.543+0000 7f14deffd700 1 -- 172.21.15.173:0/550030264 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14d8040fa0 con 0x7f14e80f6510 2024-02-19T22:01:37.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.544+0000 7f14edd45700 1 --2- 172.21.15.173:0/550030264 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14d402ff10 0x7f14d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:37.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.544+0000 7f14edd45700 1 --2- 172.21.15.173:0/550030264 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14d402ff10 0x7f14d40323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f14e40096a0 tx=0x7f14e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:37.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.548+0000 7f14deffd700 1 -- 172.21.15.173:0/550030264 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14d8014420 con 0x7f14e80f6510 2024-02-19T22:01:37.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.755+0000 7f14f07aa700 1 -- 172.21.15.173:0/550030264 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14e8108210 con 0x7f14e80f6510 2024-02-19T22:01:37.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.756+0000 7f14deffd700 1 -- 172.21.15.173:0/550030264 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f14d8010230 con 0x7f14e80f6510 2024-02-19T22:01:37.756 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:37.756 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:37.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.757+0000 7f14f07aa700 1 -- 172.21.15.173:0/550030264 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14d402ff10 msgr2=0x7f14d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:37.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.757+0000 7f14f07aa700 1 --2- 172.21.15.173:0/550030264 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14d402ff10 0x7f14d40323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f14e40096a0 tx=0x7f14e4006b40).stop 2024-02-19T22:01:37.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.757+0000 7f14f07aa700 1 -- 172.21.15.173:0/550030264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e80f6510 msgr2=0x7f14e81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:37.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.757+0000 7f14f07aa700 1 --2- 172.21.15.173:0/550030264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e80f6510 0x7f14e81067e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f14d8005010 tx=0x7f14d800b040).stop 2024-02-19T22:01:37.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.757+0000 7f14f07aa700 1 -- 172.21.15.173:0/550030264 shutdown_connections 2024-02-19T22:01:37.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.757+0000 7f14f07aa700 1 --2- 172.21.15.173:0/550030264 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f14d402ff10 0x7f14d40323d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:37.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.757+0000 7f14f07aa700 1 --2- 172.21.15.173:0/550030264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e80f6510 0x7f14e81067e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:37.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.757+0000 7f14f07aa700 1 -- 172.21.15.173:0/550030264 >> 172.21.15.173:0/550030264 conn(0x7f14e80f1c70 msgr2=0x7f14e80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:37.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.757+0000 7f14f07aa700 1 -- 172.21.15.173:0/550030264 shutdown_connections 2024-02-19T22:01:37.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:37.757+0000 7f14f07aa700 1 -- 172.21.15.173:0/550030264 wait complete. 2024-02-19T22:01:37.758 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:36.504259+0000 mgr.smithi143.bissbc (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:37.757132+0000 mon.smithi143 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.173:0/550030264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:39.500 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:39.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:38.504496+0000 mgr.smithi143.bissbc (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:41.561 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.560+0000 7efeb32bf700 1 -- 172.21.15.173:0/1109355183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeac060280 msgr2=0x7efeac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.560+0000 7efeb32bf700 1 --2- 172.21.15.173:0/1109355183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeac060280 0x7efeac0f90f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7efe9c009fb0 tx=0x7efe9c009b30).stop 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.560+0000 7efeb32bf700 1 -- 172.21.15.173:0/1109355183 shutdown_connections 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.560+0000 7efeb32bf700 1 --2- 172.21.15.173:0/1109355183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeac060280 0x7efeac0f90f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.560+0000 7efeb32bf700 1 -- 172.21.15.173:0/1109355183 >> 172.21.15.173:0/1109355183 conn(0x7efeac0f0b70 msgr2=0x7efeac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.560+0000 7efeb32bf700 1 -- 172.21.15.173:0/1109355183 shutdown_connections 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.560+0000 7efeb32bf700 1 -- 172.21.15.173:0/1109355183 wait complete. 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.561+0000 7efeb32bf700 1 Processor -- start 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.561+0000 7efeb32bf700 1 -- start start 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.561+0000 7efeb32bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeac060280 0x7efeac106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.561+0000 7efeb32bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac106c70 con 0x7efeac060280 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.561+0000 7efeb105b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeac060280 0x7efeac106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.561+0000 7efeb105b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeac060280 0x7efeac106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:59560/0 (socket says 172.21.15.173:59560) 2024-02-19T22:01:41.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.561+0000 7efeb105b700 1 -- 172.21.15.173:0/193803874 learned_addr learned my addr 172.21.15.173:0/193803874 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:41.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.561+0000 7efeb105b700 1 -- 172.21.15.173:0/193803874 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c005040 con 0x7efeac060280 2024-02-19T22:01:41.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.562+0000 7efeb105b700 1 --2- 172.21.15.173:0/193803874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeac060280 0x7efeac106780 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7efe9c0089f0 tx=0x7efe9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:41.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.562+0000 7efea1ffb700 1 -- 172.21.15.173:0/193803874 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe9c009320 con 0x7efeac060280 2024-02-19T22:01:41.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.562+0000 7efeb32bf700 1 -- 172.21.15.173:0/193803874 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeac106e70 con 0x7efeac060280 2024-02-19T22:01:41.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.562+0000 7efea1ffb700 1 -- 172.21.15.173:0/193803874 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe9c013380 con 0x7efeac060280 2024-02-19T22:01:41.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.562+0000 7efea1ffb700 1 -- 172.21.15.173:0/193803874 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe9c014e20 con 0x7efeac060280 2024-02-19T22:01:41.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.562+0000 7efeb32bf700 1 -- 172.21.15.173:0/193803874 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac107300 con 0x7efeac060280 2024-02-19T22:01:41.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.563+0000 7efea1ffb700 1 -- 172.21.15.173:0/193803874 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe9c014420 con 0x7efeac060280 2024-02-19T22:01:41.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.563+0000 7efeb32bf700 1 -- 172.21.15.173:0/193803874 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeac045ad0 con 0x7efeac060280 2024-02-19T22:01:41.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.563+0000 7efea1ffb700 1 --2- 172.21.15.173:0/193803874 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efe9802ff10 0x7efe980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:41.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.563+0000 7efea1ffb700 1 -- 172.21.15.173:0/193803874 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe9c040fb0 con 0x7efeac060280 2024-02-19T22:01:41.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.563+0000 7efeb085a700 1 --2- 172.21.15.173:0/193803874 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efe9802ff10 0x7efe980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:41.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.564+0000 7efeb085a700 1 --2- 172.21.15.173:0/193803874 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efe9802ff10 0x7efe980323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7efea80096a0 tx=0x7efea8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:41.567 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.568+0000 7efea1ffb700 1 -- 172.21.15.173:0/193803874 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe9c06d2b0 con 0x7efeac060280 2024-02-19T22:01:41.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.775+0000 7efeb32bf700 1 -- 172.21.15.173:0/193803874 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efeac1081b0 con 0x7efeac060280 2024-02-19T22:01:41.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.776+0000 7efea1ffb700 1 -- 172.21.15.173:0/193803874 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efe9c014790 con 0x7efeac060280 2024-02-19T22:01:41.776 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:41.776 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:41.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.777+0000 7efeb32bf700 1 -- 172.21.15.173:0/193803874 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efe9802ff10 msgr2=0x7efe980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:41.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.777+0000 7efeb32bf700 1 --2- 172.21.15.173:0/193803874 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efe9802ff10 0x7efe980323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7efea80096a0 tx=0x7efea8006b40).stop 2024-02-19T22:01:41.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.777+0000 7efeb32bf700 1 -- 172.21.15.173:0/193803874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeac060280 msgr2=0x7efeac106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:41.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.777+0000 7efeb32bf700 1 --2- 172.21.15.173:0/193803874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeac060280 0x7efeac106780 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7efe9c0089f0 tx=0x7efe9c00b040).stop 2024-02-19T22:01:41.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.778+0000 7efeb32bf700 1 -- 172.21.15.173:0/193803874 shutdown_connections 2024-02-19T22:01:41.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.778+0000 7efeb32bf700 1 --2- 172.21.15.173:0/193803874 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efe9802ff10 0x7efe980323d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:41.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.778+0000 7efeb32bf700 1 --2- 172.21.15.173:0/193803874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeac060280 0x7efeac106780 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:41.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.778+0000 7efeb32bf700 1 -- 172.21.15.173:0/193803874 >> 172.21.15.173:0/193803874 conn(0x7efeac0f0b70 msgr2=0x7efeac0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:41.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.778+0000 7efeb32bf700 1 -- 172.21.15.173:0/193803874 shutdown_connections 2024-02-19T22:01:41.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:41.778+0000 7efeb32bf700 1 -- 172.21.15.173:0/193803874 wait complete. 2024-02-19T22:01:41.778 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:40.504841+0000 mgr.smithi143.bissbc (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:41.777175+0000 mon.smithi143 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.173:0/193803874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:43.513 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:43.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:42.505058+0000 mgr.smithi143.bissbc (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:45.457 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.441+0000 7f4290d45700 1 -- 172.21.15.173:0/2988167068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f428c0f5410 msgr2=0x7f428c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:45.457 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.441+0000 7f4290d45700 1 --2- 172.21.15.173:0/2988167068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f428c0f5410 0x7f428c0f5830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f4280009fb0 tx=0x7f4280009b30).stop 2024-02-19T22:01:45.457 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.441+0000 7f4290d45700 1 -- 172.21.15.173:0/2988167068 shutdown_connections 2024-02-19T22:01:45.457 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.441+0000 7f4290d45700 1 --2- 172.21.15.173:0/2988167068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f428c0f5410 0x7f428c0f5830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:45.457 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.441+0000 7f4290d45700 1 -- 172.21.15.173:0/2988167068 >> 172.21.15.173:0/2988167068 conn(0x7f428c0f0b70 msgr2=0x7f428c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:45.457 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.441+0000 7f4290d45700 1 -- 172.21.15.173:0/2988167068 shutdown_connections 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.441+0000 7f4290d45700 1 -- 172.21.15.173:0/2988167068 wait complete. 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.441+0000 7f4290d45700 1 Processor -- start 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.441+0000 7f4290d45700 1 -- start start 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f4290d45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f428c0f5410 0x7f428c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f4290d45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c105bd0 con 0x7f428c0f5410 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f428a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f428c0f5410 0x7f428c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f428a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f428c0f5410 0x7f428c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:59584/0 (socket says 172.21.15.173:59584) 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f428a59c700 1 -- 172.21.15.173:0/260841745 learned_addr learned my addr 172.21.15.173:0/260841745 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f428a59c700 1 -- 172.21.15.173:0/260841745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4280005040 con 0x7f428c0f5410 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f428a59c700 1 --2- 172.21.15.173:0/260841745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f428c0f5410 0x7f428c1056e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f4280005010 tx=0x7f428000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f427affd700 1 -- 172.21.15.173:0/260841745 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4280009320 con 0x7f428c0f5410 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f4290d45700 1 -- 172.21.15.173:0/260841745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c105dd0 con 0x7f428c0f5410 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f427affd700 1 -- 172.21.15.173:0/260841745 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4280013380 con 0x7f428c0f5410 2024-02-19T22:01:45.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f427affd700 1 -- 172.21.15.173:0/260841745 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4280014e50 con 0x7f428c0f5410 2024-02-19T22:01:45.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.442+0000 7f4290d45700 1 -- 172.21.15.173:0/260841745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c106260 con 0x7f428c0f5410 2024-02-19T22:01:45.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.443+0000 7f427affd700 1 -- 172.21.15.173:0/260841745 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4280014420 con 0x7f428c0f5410 2024-02-19T22:01:45.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.443+0000 7f4290d45700 1 -- 172.21.15.173:0/260841745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428c045ad0 con 0x7f428c0f5410 2024-02-19T22:01:45.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.443+0000 7f427affd700 1 --2- 172.21.15.173:0/260841745 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f427402ff10 0x7f42740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:45.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.443+0000 7f427affd700 1 -- 172.21.15.173:0/260841745 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4280040f40 con 0x7f428c0f5410 2024-02-19T22:01:45.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.444+0000 7f4289d9b700 1 --2- 172.21.15.173:0/260841745 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f427402ff10 0x7f42740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:45.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.444+0000 7f4289d9b700 1 --2- 172.21.15.173:0/260841745 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f427402ff10 0x7f42740323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f427c0096a0 tx=0x7f427c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:45.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.448+0000 7f427affd700 1 -- 172.21.15.173:0/260841745 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42800095f0 con 0x7f428c0f5410 2024-02-19T22:01:45.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.653+0000 7f4290d45700 1 -- 172.21.15.173:0/260841745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f428c107110 con 0x7f428c0f5410 2024-02-19T22:01:45.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.654+0000 7f427affd700 1 -- 172.21.15.173:0/260841745 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f428001db70 con 0x7f428c0f5410 2024-02-19T22:01:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:45.654 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:45.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.655+0000 7f4290d45700 1 -- 172.21.15.173:0/260841745 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f427402ff10 msgr2=0x7f42740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:45.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.655+0000 7f4290d45700 1 --2- 172.21.15.173:0/260841745 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f427402ff10 0x7f42740323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f427c0096a0 tx=0x7f427c006b40).stop 2024-02-19T22:01:45.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.655+0000 7f4290d45700 1 -- 172.21.15.173:0/260841745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f428c0f5410 msgr2=0x7f428c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:45.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.655+0000 7f4290d45700 1 --2- 172.21.15.173:0/260841745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f428c0f5410 0x7f428c1056e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f4280005010 tx=0x7f428000b040).stop 2024-02-19T22:01:45.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.655+0000 7f4290d45700 1 -- 172.21.15.173:0/260841745 shutdown_connections 2024-02-19T22:01:45.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.655+0000 7f4290d45700 1 --2- 172.21.15.173:0/260841745 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f427402ff10 0x7f42740323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:45.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.655+0000 7f4290d45700 1 --2- 172.21.15.173:0/260841745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f428c0f5410 0x7f428c1056e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:45.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.655+0000 7f4290d45700 1 -- 172.21.15.173:0/260841745 >> 172.21.15.173:0/260841745 conn(0x7f428c0f0b70 msgr2=0x7f428c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:45.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.655+0000 7f4290d45700 1 -- 172.21.15.173:0/260841745 shutdown_connections 2024-02-19T22:01:45.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:45.655+0000 7f4290d45700 1 -- 172.21.15.173:0/260841745 wait complete. 2024-02-19T22:01:45.656 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:44.505264+0000 mgr.smithi143.bissbc (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:45.654815+0000 mon.smithi143 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.173:0/260841745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:47.288 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:47.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:46.505518+0000 mgr.smithi143.bissbc (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:49.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.477+0000 7f59318f1700 1 -- 172.21.15.173:0/2320691600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592c0f6510 msgr2=0x7f592c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.477+0000 7f59318f1700 1 --2- 172.21.15.173:0/2320691600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592c0f6510 0x7f592c0f6930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f5920009fb0 tx=0x7f5920009b30).stop 2024-02-19T22:01:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.477+0000 7f59318f1700 1 -- 172.21.15.173:0/2320691600 shutdown_connections 2024-02-19T22:01:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.477+0000 7f59318f1700 1 --2- 172.21.15.173:0/2320691600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592c0f6510 0x7f592c0f6930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.477+0000 7f59318f1700 1 -- 172.21.15.173:0/2320691600 >> 172.21.15.173:0/2320691600 conn(0x7f592c0f1c70 msgr2=0x7f592c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.477+0000 7f59318f1700 1 -- 172.21.15.173:0/2320691600 shutdown_connections 2024-02-19T22:01:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.477+0000 7f59318f1700 1 -- 172.21.15.173:0/2320691600 wait complete. 2024-02-19T22:01:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.478+0000 7f59318f1700 1 Processor -- start 2024-02-19T22:01:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.478+0000 7f59318f1700 1 -- start start 2024-02-19T22:01:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.478+0000 7f59318f1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592c0f6510 0x7f592c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.478+0000 7f59318f1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c106cd0 con 0x7f592c0f6510 2024-02-19T22:01:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.478+0000 7f592affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592c0f6510 0x7f592c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:49.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.478+0000 7f592affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592c0f6510 0x7f592c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:59590/0 (socket says 172.21.15.173:59590) 2024-02-19T22:01:49.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.478+0000 7f592affd700 1 -- 172.21.15.173:0/1191494069 learned_addr learned my addr 172.21.15.173:0/1191494069 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:49.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.478+0000 7f592affd700 1 -- 172.21.15.173:0/1191494069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5920005040 con 0x7f592c0f6510 2024-02-19T22:01:49.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.479+0000 7f592affd700 1 --2- 172.21.15.173:0/1191494069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592c0f6510 0x7f592c1067e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f5920005010 tx=0x7f592000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:49.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.479+0000 7f59137fe700 1 -- 172.21.15.173:0/1191494069 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5920009320 con 0x7f592c0f6510 2024-02-19T22:01:49.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.479+0000 7f59137fe700 1 -- 172.21.15.173:0/1191494069 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5920013380 con 0x7f592c0f6510 2024-02-19T22:01:49.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.479+0000 7f59318f1700 1 -- 172.21.15.173:0/1191494069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592c106ed0 con 0x7f592c0f6510 2024-02-19T22:01:49.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.479+0000 7f59137fe700 1 -- 172.21.15.173:0/1191494069 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5920014e50 con 0x7f592c0f6510 2024-02-19T22:01:49.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.479+0000 7f59318f1700 1 -- 172.21.15.173:0/1191494069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592c107360 con 0x7f592c0f6510 2024-02-19T22:01:49.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.480+0000 7f59137fe700 1 -- 172.21.15.173:0/1191494069 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59200095f0 con 0x7f592c0f6510 2024-02-19T22:01:49.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.480+0000 7f59318f1700 1 -- 172.21.15.173:0/1191494069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592c045ad0 con 0x7f592c0f6510 2024-02-19T22:01:49.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.480+0000 7f59137fe700 1 --2- 172.21.15.173:0/1191494069 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f591402ff10 0x7f59140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:49.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.480+0000 7f59137fe700 1 -- 172.21.15.173:0/1191494069 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5920040e90 con 0x7f592c0f6510 2024-02-19T22:01:49.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.480+0000 7f592a7fc700 1 --2- 172.21.15.173:0/1191494069 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f591402ff10 0x7f59140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:49.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.480+0000 7f592a7fc700 1 --2- 172.21.15.173:0/1191494069 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f591402ff10 0x7f59140323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f591c0096a0 tx=0x7f591c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:49.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.485+0000 7f59137fe700 1 -- 172.21.15.173:0/1191494069 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5920014420 con 0x7f592c0f6510 2024-02-19T22:01:49.689 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.690+0000 7f59318f1700 1 -- 172.21.15.173:0/1191494069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f592c108210 con 0x7f592c0f6510 2024-02-19T22:01:49.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.691+0000 7f59137fe700 1 -- 172.21.15.173:0/1191494069 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5920010230 con 0x7f592c0f6510 2024-02-19T22:01:49.690 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:49.690 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:49.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.692+0000 7f59318f1700 1 -- 172.21.15.173:0/1191494069 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f591402ff10 msgr2=0x7f59140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:49.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.692+0000 7f59318f1700 1 --2- 172.21.15.173:0/1191494069 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f591402ff10 0x7f59140323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f591c0096a0 tx=0x7f591c006b40).stop 2024-02-19T22:01:49.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.692+0000 7f59318f1700 1 -- 172.21.15.173:0/1191494069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592c0f6510 msgr2=0x7f592c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:49.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.692+0000 7f59318f1700 1 --2- 172.21.15.173:0/1191494069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592c0f6510 0x7f592c1067e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f5920005010 tx=0x7f592000b040).stop 2024-02-19T22:01:49.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.692+0000 7f59318f1700 1 -- 172.21.15.173:0/1191494069 shutdown_connections 2024-02-19T22:01:49.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.692+0000 7f59318f1700 1 --2- 172.21.15.173:0/1191494069 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f591402ff10 0x7f59140323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:49.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.692+0000 7f59318f1700 1 --2- 172.21.15.173:0/1191494069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592c0f6510 0x7f592c1067e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:49.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.692+0000 7f59318f1700 1 -- 172.21.15.173:0/1191494069 >> 172.21.15.173:0/1191494069 conn(0x7f592c0f1c70 msgr2=0x7f592c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:49.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.692+0000 7f59318f1700 1 -- 172.21.15.173:0/1191494069 shutdown_connections 2024-02-19T22:01:49.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:49.692+0000 7f59318f1700 1 -- 172.21.15.173:0/1191494069 wait complete. 2024-02-19T22:01:49.693 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:48.505820+0000 mgr.smithi143.bissbc (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:49.691853+0000 mon.smithi143 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.173:0/1191494069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:51.526 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:51.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:52.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:50.506144+0000 mgr.smithi143.bissbc (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:53.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.527+0000 7f6548f68700 1 -- 172.21.15.173:0/328078941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65440f62a0 msgr2=0x7f65440f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:53.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.527+0000 7f6548f68700 1 --2- 172.21.15.173:0/328078941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65440f62a0 0x7f65440f66c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f6534009fb0 tx=0x7f6534009b30).stop 2024-02-19T22:01:53.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.527+0000 7f6548f68700 1 -- 172.21.15.173:0/328078941 shutdown_connections 2024-02-19T22:01:53.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.527+0000 7f6548f68700 1 --2- 172.21.15.173:0/328078941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65440f62a0 0x7f65440f66c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:53.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.527+0000 7f6548f68700 1 -- 172.21.15.173:0/328078941 >> 172.21.15.173:0/328078941 conn(0x7f65440f1aa0 msgr2=0x7f65440f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:53.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.527+0000 7f6548f68700 1 -- 172.21.15.173:0/328078941 shutdown_connections 2024-02-19T22:01:53.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.527+0000 7f6548f68700 1 -- 172.21.15.173:0/328078941 wait complete. 2024-02-19T22:01:53.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.527+0000 7f6548f68700 1 Processor -- start 2024-02-19T22:01:53.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.527+0000 7f6548f68700 1 -- start start 2024-02-19T22:01:53.528 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.528+0000 7f6548f68700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65440f62a0 0x7f65441065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:53.528 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.528+0000 7f6548f68700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6544106ac0 con 0x7f65440f62a0 2024-02-19T22:01:53.529 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.528+0000 7f65437fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65440f62a0 0x7f65441065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:53.529 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.528+0000 7f65437fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65440f62a0 0x7f65441065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:40476/0 (socket says 172.21.15.173:40476) 2024-02-19T22:01:53.529 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.528+0000 7f65437fe700 1 -- 172.21.15.173:0/2355448354 learned_addr learned my addr 172.21.15.173:0/2355448354 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:53.529 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.528+0000 7f65437fe700 1 -- 172.21.15.173:0/2355448354 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6534005040 con 0x7f65440f62a0 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.528+0000 7f65437fe700 1 --2- 172.21.15.173:0/2355448354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65440f62a0 0x7f65441065d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f65340089f0 tx=0x7f653400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.528+0000 7f652bfff700 1 -- 172.21.15.173:0/2355448354 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6534009320 con 0x7f65440f62a0 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.528+0000 7f652bfff700 1 -- 172.21.15.173:0/2355448354 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6534013380 con 0x7f65440f62a0 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.528+0000 7f6548f68700 1 -- 172.21.15.173:0/2355448354 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6544106cc0 con 0x7f65440f62a0 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.529+0000 7f652bfff700 1 -- 172.21.15.173:0/2355448354 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6534014e20 con 0x7f65440f62a0 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.529+0000 7f6548f68700 1 -- 172.21.15.173:0/2355448354 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6544107150 con 0x7f65440f62a0 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.529+0000 7f652bfff700 1 -- 172.21.15.173:0/2355448354 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6534014420 con 0x7f65440f62a0 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.529+0000 7f6548f68700 1 -- 172.21.15.173:0/2355448354 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6544045ad0 con 0x7f65440f62a0 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.529+0000 7f652bfff700 1 --2- 172.21.15.173:0/2355448354 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f652c02ff10 0x7f652c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.529+0000 7f652bfff700 1 -- 172.21.15.173:0/2355448354 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6534040fc0 con 0x7f65440f62a0 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.530+0000 7f6542ffd700 1 --2- 172.21.15.173:0/2355448354 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f652c02ff10 0x7f652c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:53.530 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.530+0000 7f6542ffd700 1 --2- 172.21.15.173:0/2355448354 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f652c02ff10 0x7f652c0323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f65380096a0 tx=0x7f6538006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:53.534 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.535+0000 7f652bfff700 1 -- 172.21.15.173:0/2355448354 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f653406d2b0 con 0x7f65440f62a0 2024-02-19T22:01:53.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.738+0000 7f6548f68700 1 -- 172.21.15.173:0/2355448354 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6544108000 con 0x7f65440f62a0 2024-02-19T22:01:53.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.738+0000 7f652bfff700 1 -- 172.21.15.173:0/2355448354 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f65340095f0 con 0x7f65440f62a0 2024-02-19T22:01:53.738 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:53.738 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:53.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.740+0000 7f6548f68700 1 -- 172.21.15.173:0/2355448354 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f652c02ff10 msgr2=0x7f652c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:53.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.740+0000 7f6548f68700 1 --2- 172.21.15.173:0/2355448354 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f652c02ff10 0x7f652c0323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f65380096a0 tx=0x7f6538006b40).stop 2024-02-19T22:01:53.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.740+0000 7f6548f68700 1 -- 172.21.15.173:0/2355448354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65440f62a0 msgr2=0x7f65441065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:53.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.740+0000 7f6548f68700 1 --2- 172.21.15.173:0/2355448354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65440f62a0 0x7f65441065d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f65340089f0 tx=0x7f653400b040).stop 2024-02-19T22:01:53.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.740+0000 7f6548f68700 1 -- 172.21.15.173:0/2355448354 shutdown_connections 2024-02-19T22:01:53.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.740+0000 7f6548f68700 1 --2- 172.21.15.173:0/2355448354 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f652c02ff10 0x7f652c0323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:53.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.740+0000 7f6548f68700 1 --2- 172.21.15.173:0/2355448354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65440f62a0 0x7f65441065d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:53.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.740+0000 7f6548f68700 1 -- 172.21.15.173:0/2355448354 >> 172.21.15.173:0/2355448354 conn(0x7f65440f1aa0 msgr2=0x7f65440f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:53.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.740+0000 7f6548f68700 1 -- 172.21.15.173:0/2355448354 shutdown_connections 2024-02-19T22:01:53.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:53.740+0000 7f6548f68700 1 -- 172.21.15.173:0/2355448354 wait complete. 2024-02-19T22:01:53.741 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:52.506453+0000 mgr.smithi143.bissbc (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:53.739518+0000 mon.smithi143 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.173:0/2355448354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:55.666 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:55.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:54.506770+0000 mgr.smithi143.bissbc (mgr.14162) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T22:01:54.701520+0000 mgr.smithi143.bissbc (mgr.14162) 234 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi143 ... 2024-02-19T22:01:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Verifying port 9100 ... 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Non-zero exit code 1 from systemctl start ceph-38187086-cf71-11ee-95bb-87774f69a715@node-exporter.smithi143 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: systemctl: stderr Job for ceph-38187086-cf71-11ee-95bb-87774f69a715@node-exporter.smithi143.service failed because a timeout was exceeded. 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: systemctl: stderr See "systemctl status ceph-38187086-cf71-11ee-95bb-87774f69a715@node-exporter.smithi143.service" and "journalctl -xe" for details. 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Traceback (most recent call last): 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: main() 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: r = ctx.func(ctx) 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: return func(ctx) 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ports=daemon_ports) 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:01:56.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: RuntimeError: Failed command: systemctl start ceph-38187086-cf71-11ee-95bb-87774f69a715@node-exporter.smithi143 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Traceback (most recent call last): 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: yield (conn, connr) 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: code, '\n'.join(err))) 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi143 ... 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Verifying port 9100 ... 2024-02-19T22:01:56.912 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Non-zero exit code 1 from systemctl start ceph-38187086-cf71-11ee-95bb-87774f69a715@node-exporter.smithi143 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: systemctl: stderr Job for ceph-38187086-cf71-11ee-95bb-87774f69a715@node-exporter.smithi143.service failed because a timeout was exceeded. 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: systemctl: stderr See "systemctl status ceph-38187086-cf71-11ee-95bb-87774f69a715@node-exporter.smithi143.service" and "journalctl -xe" for details. 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Traceback (most recent call last): 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: main() 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: r = ctx.func(ctx) 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: return func(ctx) 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: ports=daemon_ports) 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:01:56.913 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:01:56.914 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:01:56.914 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:01:56.914 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: RuntimeError: Failed command: systemctl start ceph-38187086-cf71-11ee-95bb-87774f69a715@node-exporter.smithi143 2024-02-19T22:01:56.914 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T22:01:54.708485+0000 mgr.smithi143.bissbc (mgr.14162) 235 : cephadm [INF] Deploying daemon prometheus.smithi143 on smithi143 2024-02-19T22:01:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.671+0000 7efcaccde700 1 -- 172.21.15.173:0/985713848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efca80f62a0 msgr2=0x7efca80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.671+0000 7efcaccde700 1 --2- 172.21.15.173:0/985713848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efca80f62a0 0x7efca80f66c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7efc98009fb0 tx=0x7efc98009b30).stop 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.671+0000 7efcaccde700 1 -- 172.21.15.173:0/985713848 shutdown_connections 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.671+0000 7efcaccde700 1 --2- 172.21.15.173:0/985713848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efca80f62a0 0x7efca80f66c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.671+0000 7efcaccde700 1 -- 172.21.15.173:0/985713848 >> 172.21.15.173:0/985713848 conn(0x7efca80f1aa0 msgr2=0x7efca80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.671+0000 7efcaccde700 1 -- 172.21.15.173:0/985713848 shutdown_connections 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.671+0000 7efcaccde700 1 -- 172.21.15.173:0/985713848 wait complete. 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.672+0000 7efcaccde700 1 Processor -- start 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.672+0000 7efcaccde700 1 -- start start 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.672+0000 7efcaccde700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efca80f62a0 0x7efca81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.672+0000 7efcaccde700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca8106ac0 con 0x7efca80f62a0 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.672+0000 7efca77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efca80f62a0 0x7efca81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.672+0000 7efca77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efca80f62a0 0x7efca81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:40498/0 (socket says 172.21.15.173:40498) 2024-02-19T22:01:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.672+0000 7efca77fe700 1 -- 172.21.15.173:0/3051378315 learned_addr learned my addr 172.21.15.173:0/3051378315 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:01:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.673+0000 7efca77fe700 1 -- 172.21.15.173:0/3051378315 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc98005040 con 0x7efca80f62a0 2024-02-19T22:01:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.673+0000 7efca77fe700 1 --2- 172.21.15.173:0/3051378315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efca80f62a0 0x7efca81065d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7efc980089f0 tx=0x7efc9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.673+0000 7efc8ffff700 1 -- 172.21.15.173:0/3051378315 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc98009320 con 0x7efca80f62a0 2024-02-19T22:01:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.673+0000 7efcaccde700 1 -- 172.21.15.173:0/3051378315 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca8106cc0 con 0x7efca80f62a0 2024-02-19T22:01:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.673+0000 7efc8ffff700 1 -- 172.21.15.173:0/3051378315 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc98013380 con 0x7efca80f62a0 2024-02-19T22:01:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.673+0000 7efc8ffff700 1 -- 172.21.15.173:0/3051378315 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc98014e20 con 0x7efca80f62a0 2024-02-19T22:01:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.673+0000 7efcaccde700 1 -- 172.21.15.173:0/3051378315 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca8107150 con 0x7efca80f62a0 2024-02-19T22:01:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.674+0000 7efc8ffff700 1 -- 172.21.15.173:0/3051378315 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc98014420 con 0x7efca80f62a0 2024-02-19T22:01:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.674+0000 7efcaccde700 1 -- 172.21.15.173:0/3051378315 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca8045ad0 con 0x7efca80f62a0 2024-02-19T22:01:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.674+0000 7efc8ffff700 1 --2- 172.21.15.173:0/3051378315 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efc9002ff10 0x7efc900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.674+0000 7efc8ffff700 1 -- 172.21.15.173:0/3051378315 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc98040f00 con 0x7efca80f62a0 2024-02-19T22:01:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.674+0000 7efca6ffd700 1 --2- 172.21.15.173:0/3051378315 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efc9002ff10 0x7efc900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.675+0000 7efca6ffd700 1 --2- 172.21.15.173:0/3051378315 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efc9002ff10 0x7efc900323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7efc9c0096a0 tx=0x7efc9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.679+0000 7efc8ffff700 1 -- 172.21.15.173:0/3051378315 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc980095f0 con 0x7efca80f62a0 2024-02-19T22:01:57.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.888+0000 7efcaccde700 1 -- 172.21.15.173:0/3051378315 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efca8108000 con 0x7efca80f62a0 2024-02-19T22:01:57.888 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.888+0000 7efc8ffff700 1 -- 172.21.15.173:0/3051378315 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efc9801db70 con 0x7efca80f62a0 2024-02-19T22:01:57.888 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:01:57.888 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:57.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.890+0000 7efcaccde700 1 -- 172.21.15.173:0/3051378315 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efc9002ff10 msgr2=0x7efc900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:57.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.890+0000 7efcaccde700 1 --2- 172.21.15.173:0/3051378315 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efc9002ff10 0x7efc900323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7efc9c0096a0 tx=0x7efc9c006b40).stop 2024-02-19T22:01:57.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.890+0000 7efcaccde700 1 -- 172.21.15.173:0/3051378315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efca80f62a0 msgr2=0x7efca81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:57.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.890+0000 7efcaccde700 1 --2- 172.21.15.173:0/3051378315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efca80f62a0 0x7efca81065d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7efc980089f0 tx=0x7efc9800b040).stop 2024-02-19T22:01:57.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.890+0000 7efcaccde700 1 -- 172.21.15.173:0/3051378315 shutdown_connections 2024-02-19T22:01:57.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.890+0000 7efcaccde700 1 --2- 172.21.15.173:0/3051378315 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efc9002ff10 0x7efc900323d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:57.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.890+0000 7efcaccde700 1 --2- 172.21.15.173:0/3051378315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efca80f62a0 0x7efca81065d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:57.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.890+0000 7efcaccde700 1 -- 172.21.15.173:0/3051378315 >> 172.21.15.173:0/3051378315 conn(0x7efca80f1aa0 msgr2=0x7efca80f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:57.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.890+0000 7efcaccde700 1 -- 172.21.15.173:0/3051378315 shutdown_connections 2024-02-19T22:01:57.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:01:57.890+0000 7efcaccde700 1 -- 172.21.15.173:0/3051378315 wait complete. 2024-02-19T22:01:57.891 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:01:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:56.506993+0000 mgr.smithi143.bissbc (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:01:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:01:57.889613+0000 mon.smithi143 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.173:0/3051378315' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:59.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:59.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:01:58.507318+0000 mgr.smithi143.bissbc (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:01.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.684+0000 7fb280449700 1 -- 172.21.15.173:0/1967739035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb278060280 msgr2=0x7fb2780f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:01.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.684+0000 7fb280449700 1 --2- 172.21.15.173:0/1967739035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb278060280 0x7fb2780f90f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fb268009fb0 tx=0x7fb268009b30).stop 2024-02-19T22:02:01.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.684+0000 7fb280449700 1 -- 172.21.15.173:0/1967739035 shutdown_connections 2024-02-19T22:02:01.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.684+0000 7fb280449700 1 --2- 172.21.15.173:0/1967739035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb278060280 0x7fb2780f90f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:01.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.684+0000 7fb280449700 1 -- 172.21.15.173:0/1967739035 >> 172.21.15.173:0/1967739035 conn(0x7fb2780f0b70 msgr2=0x7fb2780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:01.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.684+0000 7fb280449700 1 -- 172.21.15.173:0/1967739035 shutdown_connections 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.684+0000 7fb280449700 1 -- 172.21.15.173:0/1967739035 wait complete. 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.685+0000 7fb280449700 1 Processor -- start 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.685+0000 7fb280449700 1 -- start start 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.685+0000 7fb280449700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb278060280 0x7fb2780f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.685+0000 7fb280449700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2780f4ff0 con 0x7fb278060280 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.685+0000 7fb27e1e5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb278060280 0x7fb2780f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.685+0000 7fb27e1e5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb278060280 0x7fb2780f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:46830/0 (socket says 172.21.15.173:46830) 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.685+0000 7fb27e1e5700 1 -- 172.21.15.173:0/2679375640 learned_addr learned my addr 172.21.15.173:0/2679375640 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.685+0000 7fb27e1e5700 1 -- 172.21.15.173:0/2679375640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb268005040 con 0x7fb278060280 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.685+0000 7fb27e1e5700 1 --2- 172.21.15.173:0/2679375640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb278060280 0x7fb2780f6b70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fb2680089f0 tx=0x7fb26800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.686+0000 7fb26effd700 1 -- 172.21.15.173:0/2679375640 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb268009320 con 0x7fb278060280 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.686+0000 7fb280449700 1 -- 172.21.15.173:0/2679375640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2780f5190 con 0x7fb278060280 2024-02-19T22:02:01.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.686+0000 7fb26effd700 1 -- 172.21.15.173:0/2679375640 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb268013380 con 0x7fb278060280 2024-02-19T22:02:01.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.686+0000 7fb26effd700 1 -- 172.21.15.173:0/2679375640 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb268014e20 con 0x7fb278060280 2024-02-19T22:02:01.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.686+0000 7fb280449700 1 -- 172.21.15.173:0/2679375640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2780f5620 con 0x7fb278060280 2024-02-19T22:02:01.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.686+0000 7fb26effd700 1 -- 172.21.15.173:0/2679375640 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2680095f0 con 0x7fb278060280 2024-02-19T22:02:01.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.686+0000 7fb280449700 1 -- 172.21.15.173:0/2679375640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb278059100 con 0x7fb278060280 2024-02-19T22:02:01.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.687+0000 7fb26effd700 1 --2- 172.21.15.173:0/2679375640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb26402ff10 0x7fb2640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:01.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.687+0000 7fb26effd700 1 -- 172.21.15.173:0/2679375640 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb268040e90 con 0x7fb278060280 2024-02-19T22:02:01.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.687+0000 7fb27d9e4700 1 --2- 172.21.15.173:0/2679375640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb26402ff10 0x7fb2640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:01.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.687+0000 7fb27d9e4700 1 --2- 172.21.15.173:0/2679375640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb26402ff10 0x7fb2640323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb2740096a0 tx=0x7fb274006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:01.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.692+0000 7fb26effd700 1 -- 172.21.15.173:0/2679375640 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb268014420 con 0x7fb278060280 2024-02-19T22:02:01.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.897+0000 7fb280449700 1 -- 172.21.15.173:0/2679375640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2780fc090 con 0x7fb278060280 2024-02-19T22:02:01.897 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.898+0000 7fb26effd700 1 -- 172.21.15.173:0/2679375640 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb26801d980 con 0x7fb278060280 2024-02-19T22:02:01.897 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:01.897 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:01.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.899+0000 7fb280449700 1 -- 172.21.15.173:0/2679375640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb26402ff10 msgr2=0x7fb2640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:01.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.899+0000 7fb280449700 1 --2- 172.21.15.173:0/2679375640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb26402ff10 0x7fb2640323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb2740096a0 tx=0x7fb274006b40).stop 2024-02-19T22:02:01.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.899+0000 7fb280449700 1 -- 172.21.15.173:0/2679375640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb278060280 msgr2=0x7fb2780f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:01.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.899+0000 7fb280449700 1 --2- 172.21.15.173:0/2679375640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb278060280 0x7fb2780f6b70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fb2680089f0 tx=0x7fb26800b040).stop 2024-02-19T22:02:01.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.899+0000 7fb280449700 1 -- 172.21.15.173:0/2679375640 shutdown_connections 2024-02-19T22:02:01.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.899+0000 7fb280449700 1 --2- 172.21.15.173:0/2679375640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb26402ff10 0x7fb2640323d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:01.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.899+0000 7fb280449700 1 --2- 172.21.15.173:0/2679375640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb278060280 0x7fb2780f6b70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:01.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.899+0000 7fb280449700 1 -- 172.21.15.173:0/2679375640 >> 172.21.15.173:0/2679375640 conn(0x7fb2780f0b70 msgr2=0x7fb2780f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:01.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.899+0000 7fb280449700 1 -- 172.21.15.173:0/2679375640 shutdown_connections 2024-02-19T22:02:01.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:01.899+0000 7fb280449700 1 -- 172.21.15.173:0/2679375640 wait complete. 2024-02-19T22:02:01.900 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:00.507618+0000 mgr.smithi143.bissbc (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:01.898665+0000 mon.smithi143 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.173:0/2679375640' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:03.609 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:03.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:02.507895+0000 mgr.smithi143.bissbc (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:05.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.703+0000 7f70b053b700 1 -- 172.21.15.173:0/3621223133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a80f6510 msgr2=0x7f70a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:05.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.703+0000 7f70b053b700 1 --2- 172.21.15.173:0/3621223133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a80f6510 0x7f70a80f6930 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f7098009fb0 tx=0x7f7098009b30).stop 2024-02-19T22:02:05.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.703+0000 7f70b053b700 1 -- 172.21.15.173:0/3621223133 shutdown_connections 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.703+0000 7f70b053b700 1 --2- 172.21.15.173:0/3621223133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a80f6510 0x7f70a80f6930 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.703+0000 7f70b053b700 1 -- 172.21.15.173:0/3621223133 >> 172.21.15.173:0/3621223133 conn(0x7f70a80f1c70 msgr2=0x7f70a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.703+0000 7f70b053b700 1 -- 172.21.15.173:0/3621223133 shutdown_connections 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.703+0000 7f70b053b700 1 -- 172.21.15.173:0/3621223133 wait complete. 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.704+0000 7f70b053b700 1 Processor -- start 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.704+0000 7f70b053b700 1 -- start start 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.704+0000 7f70b053b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a80f6510 0x7f70a806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.704+0000 7f70b053b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a8069e20 con 0x7f70a80f6510 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.704+0000 7f70ae2d7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a80f6510 0x7f70a806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.704+0000 7f70ae2d7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a80f6510 0x7f70a806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:46854/0 (socket says 172.21.15.173:46854) 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.704+0000 7f70ae2d7700 1 -- 172.21.15.173:0/2374537280 learned_addr learned my addr 172.21.15.173:0/2374537280 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.704+0000 7f70ae2d7700 1 -- 172.21.15.173:0/2374537280 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7098005040 con 0x7f70a80f6510 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.704+0000 7f70ae2d7700 1 --2- 172.21.15.173:0/2374537280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a80f6510 0x7f70a806b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f70980089f0 tx=0x7f709800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:05.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.705+0000 7f709effd700 1 -- 172.21.15.173:0/2374537280 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7098009320 con 0x7f70a80f6510 2024-02-19T22:02:05.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.705+0000 7f709effd700 1 -- 172.21.15.173:0/2374537280 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7098013380 con 0x7f70a80f6510 2024-02-19T22:02:05.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.705+0000 7f709effd700 1 -- 172.21.15.173:0/2374537280 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f709801ed80 con 0x7f70a80f6510 2024-02-19T22:02:05.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.705+0000 7f70b053b700 1 -- 172.21.15.173:0/2374537280 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a8069fc0 con 0x7f70a80f6510 2024-02-19T22:02:05.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.705+0000 7f70b053b700 1 -- 172.21.15.173:0/2374537280 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a806a450 con 0x7f70a80f6510 2024-02-19T22:02:05.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.705+0000 7f709effd700 1 -- 172.21.15.173:0/2374537280 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7098014420 con 0x7f70a80f6510 2024-02-19T22:02:05.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.706+0000 7f709effd700 1 --2- 172.21.15.173:0/2374537280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f709402ff60 0x7f7094032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:05.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.706+0000 7f709effd700 1 -- 172.21.15.173:0/2374537280 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7098040fb0 con 0x7f70a80f6510 2024-02-19T22:02:05.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.706+0000 7f70b053b700 1 -- 172.21.15.173:0/2374537280 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a8059100 con 0x7f70a80f6510 2024-02-19T22:02:05.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.706+0000 7f70adad6700 1 --2- 172.21.15.173:0/2374537280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f709402ff60 0x7f7094032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:05.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.706+0000 7f70adad6700 1 --2- 172.21.15.173:0/2374537280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f709402ff60 0x7f7094032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f70a40096a0 tx=0x7f70a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:05.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.711+0000 7f709effd700 1 -- 172.21.15.173:0/2374537280 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f709801c910 con 0x7f70a80f6510 2024-02-19T22:02:05.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.919+0000 7f70b053b700 1 -- 172.21.15.173:0/2374537280 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70a8045ad0 con 0x7f70a80f6510 2024-02-19T22:02:05.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.920+0000 7f709effd700 1 -- 172.21.15.173:0/2374537280 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f709801c320 con 0x7f70a80f6510 2024-02-19T22:02:05.919 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:05.919 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:05.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.921+0000 7f70b053b700 1 -- 172.21.15.173:0/2374537280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f709402ff60 msgr2=0x7f7094032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:05.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.921+0000 7f70b053b700 1 --2- 172.21.15.173:0/2374537280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f709402ff60 0x7f7094032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f70a40096a0 tx=0x7f70a4006b40).stop 2024-02-19T22:02:05.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.921+0000 7f70b053b700 1 -- 172.21.15.173:0/2374537280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a80f6510 msgr2=0x7f70a806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:05.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.921+0000 7f70b053b700 1 --2- 172.21.15.173:0/2374537280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a80f6510 0x7f70a806b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f70980089f0 tx=0x7f709800b040).stop 2024-02-19T22:02:05.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.921+0000 7f70b053b700 1 -- 172.21.15.173:0/2374537280 shutdown_connections 2024-02-19T22:02:05.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.921+0000 7f70b053b700 1 --2- 172.21.15.173:0/2374537280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f709402ff60 0x7f7094032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:05.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.921+0000 7f70b053b700 1 --2- 172.21.15.173:0/2374537280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a80f6510 0x7f70a806b9a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:05.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.921+0000 7f70b053b700 1 -- 172.21.15.173:0/2374537280 >> 172.21.15.173:0/2374537280 conn(0x7f70a80f1c70 msgr2=0x7f70a80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:05.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.921+0000 7f70b053b700 1 -- 172.21.15.173:0/2374537280 shutdown_connections 2024-02-19T22:02:05.922 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:05.921+0000 7f70b053b700 1 -- 172.21.15.173:0/2374537280 wait complete. 2024-02-19T22:02:05.922 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:04.508194+0000 mgr.smithi143.bissbc (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:05.920904+0000 mon.smithi143 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.173:0/2374537280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:07.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:07.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:06.508436+0000 mgr.smithi143.bissbc (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:09.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.818+0000 7fc37af1f700 1 -- 172.21.15.173:0/4188745692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3740f6510 msgr2=0x7fc3740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:09.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.818+0000 7fc37af1f700 1 --2- 172.21.15.173:0/4188745692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3740f6510 0x7fc3740f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fc368009fb0 tx=0x7fc368009b30).stop 2024-02-19T22:02:09.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.818+0000 7fc37af1f700 1 -- 172.21.15.173:0/4188745692 shutdown_connections 2024-02-19T22:02:09.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.818+0000 7fc37af1f700 1 --2- 172.21.15.173:0/4188745692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3740f6510 0x7fc3740f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.818+0000 7fc37af1f700 1 -- 172.21.15.173:0/4188745692 >> 172.21.15.173:0/4188745692 conn(0x7fc3740f1c70 msgr2=0x7fc3740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.818+0000 7fc37af1f700 1 -- 172.21.15.173:0/4188745692 shutdown_connections 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.818+0000 7fc37af1f700 1 -- 172.21.15.173:0/4188745692 wait complete. 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.818+0000 7fc37af1f700 1 Processor -- start 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.819+0000 7fc37af1f700 1 -- start start 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.819+0000 7fc37af1f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3740f6510 0x7fc3741067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.819+0000 7fc37af1f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc374106cd0 con 0x7fc3740f6510 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.819+0000 7fc378cbb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3740f6510 0x7fc3741067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.819+0000 7fc378cbb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3740f6510 0x7fc3741067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:46872/0 (socket says 172.21.15.173:46872) 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.819+0000 7fc378cbb700 1 -- 172.21.15.173:0/1423965242 learned_addr learned my addr 172.21.15.173:0/1423965242 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.819+0000 7fc378cbb700 1 -- 172.21.15.173:0/1423965242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc368005040 con 0x7fc3740f6510 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.819+0000 7fc378cbb700 1 --2- 172.21.15.173:0/1423965242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3740f6510 0x7fc3741067e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fc368005010 tx=0x7fc36800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.820+0000 7fc3717fa700 1 -- 172.21.15.173:0/1423965242 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc368009320 con 0x7fc3740f6510 2024-02-19T22:02:09.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.820+0000 7fc37af1f700 1 -- 172.21.15.173:0/1423965242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc374106ed0 con 0x7fc3740f6510 2024-02-19T22:02:09.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.820+0000 7fc3717fa700 1 -- 172.21.15.173:0/1423965242 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc368013380 con 0x7fc3740f6510 2024-02-19T22:02:09.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.820+0000 7fc3717fa700 1 -- 172.21.15.173:0/1423965242 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc368014e50 con 0x7fc3740f6510 2024-02-19T22:02:09.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.820+0000 7fc37af1f700 1 -- 172.21.15.173:0/1423965242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc374107360 con 0x7fc3740f6510 2024-02-19T22:02:09.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.821+0000 7fc3717fa700 1 -- 172.21.15.173:0/1423965242 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3680263a0 con 0x7fc3740f6510 2024-02-19T22:02:09.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.821+0000 7fc37af1f700 1 -- 172.21.15.173:0/1423965242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc374045ad0 con 0x7fc3740f6510 2024-02-19T22:02:09.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.821+0000 7fc3717fa700 1 --2- 172.21.15.173:0/1423965242 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc36402ff10 0x7fc3640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:09.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.821+0000 7fc3717fa700 1 -- 172.21.15.173:0/1423965242 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc368041150 con 0x7fc3740f6510 2024-02-19T22:02:09.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.821+0000 7fc373fff700 1 --2- 172.21.15.173:0/1423965242 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc36402ff10 0x7fc3640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:09.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.821+0000 7fc373fff700 1 --2- 172.21.15.173:0/1423965242 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc36402ff10 0x7fc3640323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fc3600096a0 tx=0x7fc360006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:09.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:09.826+0000 7fc3717fa700 1 -- 172.21.15.173:0/1423965242 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc36801ccf0 con 0x7fc3740f6510 2024-02-19T22:02:09.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:09 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:08.508677+0000 mgr.smithi143.bissbc (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:10.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.034+0000 7fc37af1f700 1 -- 172.21.15.173:0/1423965242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc374108210 con 0x7fc3740f6510 2024-02-19T22:02:10.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.035+0000 7fc3717fa700 1 -- 172.21.15.173:0/1423965242 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc36801cb10 con 0x7fc3740f6510 2024-02-19T22:02:10.035 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:10.035 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:10.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.037+0000 7fc37af1f700 1 -- 172.21.15.173:0/1423965242 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc36402ff10 msgr2=0x7fc3640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:10.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.037+0000 7fc37af1f700 1 --2- 172.21.15.173:0/1423965242 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc36402ff10 0x7fc3640323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fc3600096a0 tx=0x7fc360006b40).stop 2024-02-19T22:02:10.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.037+0000 7fc37af1f700 1 -- 172.21.15.173:0/1423965242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3740f6510 msgr2=0x7fc3741067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:10.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.037+0000 7fc37af1f700 1 --2- 172.21.15.173:0/1423965242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3740f6510 0x7fc3741067e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fc368005010 tx=0x7fc36800b040).stop 2024-02-19T22:02:10.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.037+0000 7fc37af1f700 1 -- 172.21.15.173:0/1423965242 shutdown_connections 2024-02-19T22:02:10.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.037+0000 7fc37af1f700 1 --2- 172.21.15.173:0/1423965242 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc36402ff10 0x7fc3640323d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:10.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.037+0000 7fc37af1f700 1 --2- 172.21.15.173:0/1423965242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3740f6510 0x7fc3741067e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:10.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.037+0000 7fc37af1f700 1 -- 172.21.15.173:0/1423965242 >> 172.21.15.173:0/1423965242 conn(0x7fc3740f1c70 msgr2=0x7fc3740f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:10.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.037+0000 7fc37af1f700 1 -- 172.21.15.173:0/1423965242 shutdown_connections 2024-02-19T22:02:10.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:10.037+0000 7fc37af1f700 1 -- 172.21.15.173:0/1423965242 wait complete. 2024-02-19T22:02:10.038 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:10.909 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:10.036402+0000 mon.smithi143 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.173:0/1423965242' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:11.809 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:11.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:11.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:11 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:10.508900+0000 mgr.smithi143.bissbc (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:13.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.889+0000 7fa1c9d95700 1 -- 172.21.15.173:0/3624280103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c40f5410 msgr2=0x7fa1c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:13.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.889+0000 7fa1c9d95700 1 --2- 172.21.15.173:0/3624280103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c40f5410 0x7fa1c40f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fa1b4009fb0 tx=0x7fa1b4009b30).stop 2024-02-19T22:02:13.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.889+0000 7fa1c9d95700 1 -- 172.21.15.173:0/3624280103 shutdown_connections 2024-02-19T22:02:13.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.889+0000 7fa1c9d95700 1 --2- 172.21.15.173:0/3624280103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c40f5410 0x7fa1c40f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:13.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.889+0000 7fa1c9d95700 1 -- 172.21.15.173:0/3624280103 >> 172.21.15.173:0/3624280103 conn(0x7fa1c40f0b70 msgr2=0x7fa1c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:13.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.889+0000 7fa1c9d95700 1 -- 172.21.15.173:0/3624280103 shutdown_connections 2024-02-19T22:02:13.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.889+0000 7fa1c9d95700 1 -- 172.21.15.173:0/3624280103 wait complete. 2024-02-19T22:02:13.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.889+0000 7fa1c9d95700 1 Processor -- start 2024-02-19T22:02:13.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.890+0000 7fa1c9d95700 1 -- start start 2024-02-19T22:02:13.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.890+0000 7fa1c9d95700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c40f5410 0x7fa1c4109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:13.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.890+0000 7fa1c9d95700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c4060220 con 0x7fa1c40f5410 2024-02-19T22:02:13.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.890+0000 7fa1c37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c40f5410 0x7fa1c4109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:13.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.890+0000 7fa1c37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c40f5410 0x7fa1c4109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:42006/0 (socket says 172.21.15.173:42006) 2024-02-19T22:02:13.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.890+0000 7fa1c37fe700 1 -- 172.21.15.173:0/1763221985 learned_addr learned my addr 172.21.15.173:0/1763221985 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:13.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.891+0000 7fa1c37fe700 1 -- 172.21.15.173:0/1763221985 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b4005040 con 0x7fa1c40f5410 2024-02-19T22:02:13.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.891+0000 7fa1c37fe700 1 --2- 172.21.15.173:0/1763221985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c40f5410 0x7fa1c4109500 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fa1b4005010 tx=0x7fa1b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:13.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.891+0000 7fa1abfff700 1 -- 172.21.15.173:0/1763221985 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1b4009320 con 0x7fa1c40f5410 2024-02-19T22:02:13.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.891+0000 7fa1abfff700 1 -- 172.21.15.173:0/1763221985 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1b4013380 con 0x7fa1c40f5410 2024-02-19T22:02:13.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.891+0000 7fa1c9d95700 1 -- 172.21.15.173:0/1763221985 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c4060420 con 0x7fa1c40f5410 2024-02-19T22:02:13.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.891+0000 7fa1abfff700 1 -- 172.21.15.173:0/1763221985 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1b4014e50 con 0x7fa1c40f5410 2024-02-19T22:02:13.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.891+0000 7fa1c9d95700 1 -- 172.21.15.173:0/1763221985 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c4109bb0 con 0x7fa1c40f5410 2024-02-19T22:02:13.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.892+0000 7fa1abfff700 1 -- 172.21.15.173:0/1763221985 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1b40095f0 con 0x7fa1c40f5410 2024-02-19T22:02:13.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.892+0000 7fa1abfff700 1 --2- 172.21.15.173:0/1763221985 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa1ac02ff10 0x7fa1ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:13.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.892+0000 7fa1c9d95700 1 -- 172.21.15.173:0/1763221985 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c4059100 con 0x7fa1c40f5410 2024-02-19T22:02:13.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.892+0000 7fa1abfff700 1 -- 172.21.15.173:0/1763221985 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1b4040e60 con 0x7fa1c40f5410 2024-02-19T22:02:13.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.892+0000 7fa1c2ffd700 1 --2- 172.21.15.173:0/1763221985 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa1ac02ff10 0x7fa1ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:13.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.892+0000 7fa1c2ffd700 1 --2- 172.21.15.173:0/1763221985 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa1ac02ff10 0x7fa1ac0323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa1b80096a0 tx=0x7fa1b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:13.896 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:13.897+0000 7fa1abfff700 1 -- 172.21.15.173:0/1763221985 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1b4014420 con 0x7fa1c40f5410 2024-02-19T22:02:14.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.106+0000 7fa1c9d95700 1 -- 172.21.15.173:0/1763221985 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1c40fafa0 con 0x7fa1c40f5410 2024-02-19T22:02:14.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.106+0000 7fa1abfff700 1 -- 172.21.15.173:0/1763221985 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa1b402b030 con 0x7fa1c40f5410 2024-02-19T22:02:14.106 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:14.106 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:14.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.108+0000 7fa1c9d95700 1 -- 172.21.15.173:0/1763221985 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa1ac02ff10 msgr2=0x7fa1ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:14.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.108+0000 7fa1c9d95700 1 --2- 172.21.15.173:0/1763221985 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa1ac02ff10 0x7fa1ac0323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa1b80096a0 tx=0x7fa1b8006b40).stop 2024-02-19T22:02:14.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.108+0000 7fa1c9d95700 1 -- 172.21.15.173:0/1763221985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c40f5410 msgr2=0x7fa1c4109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:14.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.108+0000 7fa1c9d95700 1 --2- 172.21.15.173:0/1763221985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c40f5410 0x7fa1c4109500 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fa1b4005010 tx=0x7fa1b400b040).stop 2024-02-19T22:02:14.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.108+0000 7fa1c9d95700 1 -- 172.21.15.173:0/1763221985 shutdown_connections 2024-02-19T22:02:14.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.108+0000 7fa1c9d95700 1 --2- 172.21.15.173:0/1763221985 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa1ac02ff10 0x7fa1ac0323d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:14.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.108+0000 7fa1c9d95700 1 --2- 172.21.15.173:0/1763221985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c40f5410 0x7fa1c4109500 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:14.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.108+0000 7fa1c9d95700 1 -- 172.21.15.173:0/1763221985 >> 172.21.15.173:0/1763221985 conn(0x7fa1c40f0b70 msgr2=0x7fa1c40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:14.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.108+0000 7fa1c9d95700 1 -- 172.21.15.173:0/1763221985 shutdown_connections 2024-02-19T22:02:14.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:14.108+0000 7fa1c9d95700 1 -- 172.21.15.173:0/1763221985 wait complete. 2024-02-19T22:02:14.109 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:12.509278+0000 mgr.smithi143.bissbc (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:14.107531+0000 mon.smithi143 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.173:0/1763221985' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:15.900 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:15.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:14.509658+0000 mgr.smithi143.bissbc (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:17.910 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.907+0000 7f852aeb5700 1 -- 172.21.15.173:0/559171688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85240f51a0 msgr2=0x7f85240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:17.910 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.907+0000 7f852aeb5700 1 --2- 172.21.15.173:0/559171688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85240f51a0 0x7f85240f55c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f8518009fb0 tx=0x7f8518009b30).stop 2024-02-19T22:02:17.910 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.907+0000 7f852aeb5700 1 -- 172.21.15.173:0/559171688 shutdown_connections 2024-02-19T22:02:17.910 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.907+0000 7f852aeb5700 1 --2- 172.21.15.173:0/559171688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85240f51a0 0x7f85240f55c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:17.910 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.907+0000 7f852aeb5700 1 -- 172.21.15.173:0/559171688 >> 172.21.15.173:0/559171688 conn(0x7f85240f09a0 msgr2=0x7f85240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:17.910 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.907+0000 7f852aeb5700 1 -- 172.21.15.173:0/559171688 shutdown_connections 2024-02-19T22:02:17.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.907+0000 7f852aeb5700 1 -- 172.21.15.173:0/559171688 wait complete. 2024-02-19T22:02:17.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.907+0000 7f852aeb5700 1 Processor -- start 2024-02-19T22:02:17.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.907+0000 7f852aeb5700 1 -- start start 2024-02-19T22:02:17.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.907+0000 7f852aeb5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85240f51a0 0x7f85241092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:17.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.907+0000 7f852aeb5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8524060220 con 0x7f85240f51a0 2024-02-19T22:02:17.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.908+0000 7f8529eb3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85240f51a0 0x7f85241092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:17.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.908+0000 7f8529eb3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85240f51a0 0x7f85241092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:42028/0 (socket says 172.21.15.173:42028) 2024-02-19T22:02:17.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.908+0000 7f8529eb3700 1 -- 172.21.15.173:0/554600529 learned_addr learned my addr 172.21.15.173:0/554600529 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:17.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.908+0000 7f8529eb3700 1 -- 172.21.15.173:0/554600529 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8518005040 con 0x7f85240f51a0 2024-02-19T22:02:17.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.908+0000 7f8529eb3700 1 --2- 172.21.15.173:0/554600529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85240f51a0 0x7f85241092f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f85180089f0 tx=0x7f851800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:17.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.908+0000 7f85167fc700 1 -- 172.21.15.173:0/554600529 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8518009320 con 0x7f85240f51a0 2024-02-19T22:02:17.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.908+0000 7f852aeb5700 1 -- 172.21.15.173:0/554600529 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8524060420 con 0x7f85240f51a0 2024-02-19T22:02:17.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.908+0000 7f85167fc700 1 -- 172.21.15.173:0/554600529 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8518013380 con 0x7f85240f51a0 2024-02-19T22:02:17.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.908+0000 7f85167fc700 1 -- 172.21.15.173:0/554600529 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8518014e20 con 0x7f85240f51a0 2024-02-19T22:02:17.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.908+0000 7f852aeb5700 1 -- 172.21.15.173:0/554600529 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85241099a0 con 0x7f85240f51a0 2024-02-19T22:02:17.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.909+0000 7f85167fc700 1 -- 172.21.15.173:0/554600529 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85180095f0 con 0x7f85240f51a0 2024-02-19T22:02:17.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.909+0000 7f852aeb5700 1 -- 172.21.15.173:0/554600529 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8524059100 con 0x7f85240f51a0 2024-02-19T22:02:17.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.909+0000 7f85167fc700 1 --2- 172.21.15.173:0/554600529 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f851002ff10 0x7f85100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:17.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.909+0000 7f85167fc700 1 -- 172.21.15.173:0/554600529 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8518040f60 con 0x7f85240f51a0 2024-02-19T22:02:17.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.909+0000 7f85296b2700 1 --2- 172.21.15.173:0/554600529 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f851002ff10 0x7f85100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:17.913 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.910+0000 7f85296b2700 1 --2- 172.21.15.173:0/554600529 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f851002ff10 0x7f85100323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f85200096a0 tx=0x7f8520006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:17.915 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:17.915+0000 7f85167fc700 1 -- 172.21.15.173:0/554600529 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f851801db60 con 0x7f85240f51a0 2024-02-19T22:02:18.121 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.121+0000 7f852aeb5700 1 -- 172.21.15.173:0/554600529 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8524045ad0 con 0x7f85240f51a0 2024-02-19T22:02:18.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.121+0000 7f85167fc700 1 -- 172.21.15.173:0/554600529 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f851801d980 con 0x7f85240f51a0 2024-02-19T22:02:18.122 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:18.122 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:18.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.124+0000 7f852aeb5700 1 -- 172.21.15.173:0/554600529 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f851002ff10 msgr2=0x7f85100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:18.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.124+0000 7f852aeb5700 1 --2- 172.21.15.173:0/554600529 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f851002ff10 0x7f85100323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f85200096a0 tx=0x7f8520006b40).stop 2024-02-19T22:02:18.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.124+0000 7f852aeb5700 1 -- 172.21.15.173:0/554600529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85240f51a0 msgr2=0x7f85241092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:18.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.124+0000 7f852aeb5700 1 --2- 172.21.15.173:0/554600529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85240f51a0 0x7f85241092f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f85180089f0 tx=0x7f851800b040).stop 2024-02-19T22:02:18.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.124+0000 7f852aeb5700 1 -- 172.21.15.173:0/554600529 shutdown_connections 2024-02-19T22:02:18.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.124+0000 7f852aeb5700 1 --2- 172.21.15.173:0/554600529 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f851002ff10 0x7f85100323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:18.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.124+0000 7f852aeb5700 1 --2- 172.21.15.173:0/554600529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85240f51a0 0x7f85241092f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:18.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.124+0000 7f852aeb5700 1 -- 172.21.15.173:0/554600529 >> 172.21.15.173:0/554600529 conn(0x7f85240f09a0 msgr2=0x7f85240f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:18.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.124+0000 7f852aeb5700 1 -- 172.21.15.173:0/554600529 shutdown_connections 2024-02-19T22:02:18.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:18.124+0000 7f852aeb5700 1 -- 172.21.15.173:0/554600529 wait complete. 2024-02-19T22:02:18.125 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:18.909 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:16.509963+0000 mgr.smithi143.bissbc (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:18.123768+0000 mon.smithi143 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.173:0/554600529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:19.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:19.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:18.510251+0000 mgr.smithi143.bissbc (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:20.570048+0000 mon.smithi143 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T22:02:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:20.570259+0000 mon.smithi143 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:02:22.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.010+0000 7f8a8b059700 1 -- 172.21.15.173:0/4127341891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a840f6510 msgr2=0x7f8a840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:22.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.010+0000 7f8a8b059700 1 --2- 172.21.15.173:0/4127341891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a840f6510 0x7f8a840f6930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f8a78009fb0 tx=0x7f8a78009b30).stop 2024-02-19T22:02:22.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.010+0000 7f8a8b059700 1 -- 172.21.15.173:0/4127341891 shutdown_connections 2024-02-19T22:02:22.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.010+0000 7f8a8b059700 1 --2- 172.21.15.173:0/4127341891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a840f6510 0x7f8a840f6930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:22.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.010+0000 7f8a8b059700 1 -- 172.21.15.173:0/4127341891 >> 172.21.15.173:0/4127341891 conn(0x7f8a840f1c70 msgr2=0x7f8a840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:22.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.010+0000 7f8a8b059700 1 -- 172.21.15.173:0/4127341891 shutdown_connections 2024-02-19T22:02:22.010 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.010+0000 7f8a8b059700 1 -- 172.21.15.173:0/4127341891 wait complete. 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.010+0000 7f8a8b059700 1 Processor -- start 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.011+0000 7f8a8b059700 1 -- start start 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.011+0000 7f8a8b059700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a840f6510 0x7f8a841067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.011+0000 7f8a8b059700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a84106cd0 con 0x7f8a840f6510 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.011+0000 7f8a88df5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a840f6510 0x7f8a841067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.011+0000 7f8a88df5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a840f6510 0x7f8a841067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:33506/0 (socket says 172.21.15.173:33506) 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.011+0000 7f8a88df5700 1 -- 172.21.15.173:0/576443795 learned_addr learned my addr 172.21.15.173:0/576443795 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.012+0000 7f8a88df5700 1 -- 172.21.15.173:0/576443795 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78005040 con 0x7f8a840f6510 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.012+0000 7f8a88df5700 1 --2- 172.21.15.173:0/576443795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a840f6510 0x7f8a841067e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f8a780089f0 tx=0x7f8a7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.012+0000 7f8a817fa700 1 -- 172.21.15.173:0/576443795 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a78009320 con 0x7f8a840f6510 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.012+0000 7f8a8b059700 1 -- 172.21.15.173:0/576443795 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a84106ed0 con 0x7f8a840f6510 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.012+0000 7f8a817fa700 1 -- 172.21.15.173:0/576443795 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a78013380 con 0x7f8a840f6510 2024-02-19T22:02:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.012+0000 7f8a817fa700 1 -- 172.21.15.173:0/576443795 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a7801ed80 con 0x7f8a840f6510 2024-02-19T22:02:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.012+0000 7f8a8b059700 1 -- 172.21.15.173:0/576443795 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a84107360 con 0x7f8a840f6510 2024-02-19T22:02:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.013+0000 7f8a817fa700 1 -- 172.21.15.173:0/576443795 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a7801e420 con 0x7f8a840f6510 2024-02-19T22:02:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.013+0000 7f8a8b059700 1 -- 172.21.15.173:0/576443795 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a84045ad0 con 0x7f8a840f6510 2024-02-19T22:02:22.014 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.013+0000 7f8a817fa700 1 --2- 172.21.15.173:0/576443795 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a7402ff60 0x7f8a74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:22.014 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.013+0000 7f8a817fa700 1 -- 172.21.15.173:0/576443795 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a78040f30 con 0x7f8a840f6510 2024-02-19T22:02:22.014 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.013+0000 7f8a83fff700 1 --2- 172.21.15.173:0/576443795 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a7402ff60 0x7f8a74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:22.014 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.014+0000 7f8a83fff700 1 --2- 172.21.15.173:0/576443795 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a7402ff60 0x7f8a74032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:22.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.018+0000 7f8a817fa700 1 -- 172.21.15.173:0/576443795 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a7800e2a0 con 0x7f8a840f6510 2024-02-19T22:02:22.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.223+0000 7f8a8b059700 1 -- 172.21.15.173:0/576443795 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a84108210 con 0x7f8a840f6510 2024-02-19T22:02:22.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.224+0000 7f8a817fa700 1 -- 172.21.15.173:0/576443795 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8a7800e2a0 con 0x7f8a840f6510 2024-02-19T22:02:22.223 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:22.223 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:22.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.225+0000 7f8a8b059700 1 -- 172.21.15.173:0/576443795 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a7402ff60 msgr2=0x7f8a74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:22.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.225+0000 7f8a8b059700 1 --2- 172.21.15.173:0/576443795 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a7402ff60 0x7f8a74032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).stop 2024-02-19T22:02:22.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.225+0000 7f8a8b059700 1 -- 172.21.15.173:0/576443795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a840f6510 msgr2=0x7f8a841067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:22.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.225+0000 7f8a8b059700 1 --2- 172.21.15.173:0/576443795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a840f6510 0x7f8a841067e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f8a780089f0 tx=0x7f8a7800b040).stop 2024-02-19T22:02:22.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.225+0000 7f8a8b059700 1 -- 172.21.15.173:0/576443795 shutdown_connections 2024-02-19T22:02:22.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.225+0000 7f8a8b059700 1 --2- 172.21.15.173:0/576443795 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a7402ff60 0x7f8a74032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:22.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.225+0000 7f8a8b059700 1 --2- 172.21.15.173:0/576443795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a840f6510 0x7f8a841067e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:22.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.226+0000 7f8a8b059700 1 -- 172.21.15.173:0/576443795 >> 172.21.15.173:0/576443795 conn(0x7f8a840f1c70 msgr2=0x7f8a840f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:22.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.226+0000 7f8a8b059700 1 -- 172.21.15.173:0/576443795 shutdown_connections 2024-02-19T22:02:22.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:22.226+0000 7f8a8b059700 1 -- 172.21.15.173:0/576443795 wait complete. 2024-02-19T22:02:22.226 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:20.510554+0000 mgr.smithi143.bissbc (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:22.224607+0000 mon.smithi143 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.173:0/576443795' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:23.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:23.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:24.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:22.510846+0000 mgr.smithi143.bissbc (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:26.172 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.172+0000 7f053d4c0700 1 -- 172.21.15.173:0/2161857707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05380f47c0 msgr2=0x7f05380f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:26.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.172+0000 7f053d4c0700 1 --2- 172.21.15.173:0/2161857707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05380f47c0 0x7f05380f4be0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f052c009fb0 tx=0x7f052c009b30).stop 2024-02-19T22:02:26.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.172+0000 7f053d4c0700 1 -- 172.21.15.173:0/2161857707 shutdown_connections 2024-02-19T22:02:26.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.172+0000 7f053d4c0700 1 --2- 172.21.15.173:0/2161857707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05380f47c0 0x7f05380f4be0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:26.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.172+0000 7f053d4c0700 1 -- 172.21.15.173:0/2161857707 >> 172.21.15.173:0/2161857707 conn(0x7f05380f02e0 msgr2=0x7f05380f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:26.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.172+0000 7f053d4c0700 1 -- 172.21.15.173:0/2161857707 shutdown_connections 2024-02-19T22:02:26.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.172+0000 7f053d4c0700 1 -- 172.21.15.173:0/2161857707 wait complete. 2024-02-19T22:02:26.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.173+0000 7f053d4c0700 1 Processor -- start 2024-02-19T22:02:26.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.173+0000 7f053d4c0700 1 -- start start 2024-02-19T22:02:26.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.173+0000 7f053d4c0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05380f47c0 0x7f05381067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:26.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.173+0000 7f053d4c0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0538106cc0 con 0x7f05380f47c0 2024-02-19T22:02:26.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.173+0000 7f0536ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05380f47c0 0x7f05381067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:26.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.173+0000 7f0536ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05380f47c0 0x7f05381067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:33528/0 (socket says 172.21.15.173:33528) 2024-02-19T22:02:26.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.173+0000 7f0536ffd700 1 -- 172.21.15.173:0/1940189007 learned_addr learned my addr 172.21.15.173:0/1940189007 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:26.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.174+0000 7f0536ffd700 1 -- 172.21.15.173:0/1940189007 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c005040 con 0x7f05380f47c0 2024-02-19T22:02:26.175 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.174+0000 7f0536ffd700 1 --2- 172.21.15.173:0/1940189007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05380f47c0 0x7f05381067d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f052c005010 tx=0x7f052c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:26.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.174+0000 7f051f7fe700 1 -- 172.21.15.173:0/1940189007 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f052c009320 con 0x7f05380f47c0 2024-02-19T22:02:26.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.174+0000 7f051f7fe700 1 -- 172.21.15.173:0/1940189007 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f052c013380 con 0x7f05380f47c0 2024-02-19T22:02:26.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.174+0000 7f053d4c0700 1 -- 172.21.15.173:0/1940189007 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0538106ec0 con 0x7f05380f47c0 2024-02-19T22:02:26.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.174+0000 7f051f7fe700 1 -- 172.21.15.173:0/1940189007 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f052c014e50 con 0x7f05380f47c0 2024-02-19T22:02:26.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.174+0000 7f053d4c0700 1 -- 172.21.15.173:0/1940189007 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0538107350 con 0x7f05380f47c0 2024-02-19T22:02:26.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.175+0000 7f051f7fe700 1 -- 172.21.15.173:0/1940189007 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f052c0263a0 con 0x7f05380f47c0 2024-02-19T22:02:26.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.175+0000 7f051f7fe700 1 --2- 172.21.15.173:0/1940189007 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f052002ff10 0x7f05200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:26.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.175+0000 7f051f7fe700 1 -- 172.21.15.173:0/1940189007 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f052c040e70 con 0x7f05380f47c0 2024-02-19T22:02:26.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.175+0000 7f053d4c0700 1 -- 172.21.15.173:0/1940189007 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0538045ad0 con 0x7f05380f47c0 2024-02-19T22:02:26.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.175+0000 7f05367fc700 1 --2- 172.21.15.173:0/1940189007 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f052002ff10 0x7f05200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:26.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.175+0000 7f05367fc700 1 --2- 172.21.15.173:0/1940189007 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f052002ff10 0x7f05200323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f05280096a0 tx=0x7f0528006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:26.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.180+0000 7f051f7fe700 1 -- 172.21.15.173:0/1940189007 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f052c012920 con 0x7f05380f47c0 2024-02-19T22:02:26.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.388+0000 7f053d4c0700 1 -- 172.21.15.173:0/1940189007 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05380fc090 con 0x7f05380f47c0 2024-02-19T22:02:26.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.389+0000 7f051f7fe700 1 -- 172.21.15.173:0/1940189007 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f052c02b020 con 0x7f05380f47c0 2024-02-19T22:02:26.388 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:26.388 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:26.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.390+0000 7f053d4c0700 1 -- 172.21.15.173:0/1940189007 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f052002ff10 msgr2=0x7f05200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:26.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.390+0000 7f053d4c0700 1 --2- 172.21.15.173:0/1940189007 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f052002ff10 0x7f05200323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f05280096a0 tx=0x7f0528006b40).stop 2024-02-19T22:02:26.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.390+0000 7f053d4c0700 1 -- 172.21.15.173:0/1940189007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05380f47c0 msgr2=0x7f05381067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:26.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.390+0000 7f053d4c0700 1 --2- 172.21.15.173:0/1940189007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05380f47c0 0x7f05381067d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f052c005010 tx=0x7f052c00b040).stop 2024-02-19T22:02:26.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.390+0000 7f053d4c0700 1 -- 172.21.15.173:0/1940189007 shutdown_connections 2024-02-19T22:02:26.390 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.390+0000 7f053d4c0700 1 --2- 172.21.15.173:0/1940189007 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f052002ff10 0x7f05200323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:26.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.390+0000 7f053d4c0700 1 --2- 172.21.15.173:0/1940189007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05380f47c0 0x7f05381067d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:26.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.390+0000 7f053d4c0700 1 -- 172.21.15.173:0/1940189007 >> 172.21.15.173:0/1940189007 conn(0x7f05380f02e0 msgr2=0x7f05380f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:26.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.390+0000 7f053d4c0700 1 -- 172.21.15.173:0/1940189007 shutdown_connections 2024-02-19T22:02:26.391 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:26.390+0000 7f053d4c0700 1 -- 172.21.15.173:0/1940189007 wait complete. 2024-02-19T22:02:26.391 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:24.511089+0000 mgr.smithi143.bissbc (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:26.389980+0000 mon.smithi143 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.173:0/1940189007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:28.251 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:28.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:26.511341+0000 mgr.smithi143.bissbc (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:30.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.151+0000 7ff3f4418700 1 -- 172.21.15.173:0/2004584202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec0f6510 msgr2=0x7ff3ec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:30.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.151+0000 7ff3f4418700 1 --2- 172.21.15.173:0/2004584202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec0f6510 0x7ff3ec0f6930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7ff3dc009fb0 tx=0x7ff3dc009b30).stop 2024-02-19T22:02:30.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.151+0000 7ff3f4418700 1 -- 172.21.15.173:0/2004584202 shutdown_connections 2024-02-19T22:02:30.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.151+0000 7ff3f4418700 1 --2- 172.21.15.173:0/2004584202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec0f6510 0x7ff3ec0f6930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:30.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.151+0000 7ff3f4418700 1 -- 172.21.15.173:0/2004584202 >> 172.21.15.173:0/2004584202 conn(0x7ff3ec0f1c70 msgr2=0x7ff3ec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:30.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.151+0000 7ff3f4418700 1 -- 172.21.15.173:0/2004584202 shutdown_connections 2024-02-19T22:02:30.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.151+0000 7ff3f4418700 1 -- 172.21.15.173:0/2004584202 wait complete. 2024-02-19T22:02:30.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.152+0000 7ff3f4418700 1 Processor -- start 2024-02-19T22:02:30.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.152+0000 7ff3f4418700 1 -- start start 2024-02-19T22:02:30.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.152+0000 7ff3f4418700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec0f6510 0x7ff3ec1089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.152+0000 7ff3f4418700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec108eb0 con 0x7ff3ec0f6510 2024-02-19T22:02:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.152+0000 7ff3f21b4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec0f6510 0x7ff3ec1089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.152+0000 7ff3f21b4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec0f6510 0x7ff3ec1089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:33540/0 (socket says 172.21.15.173:33540) 2024-02-19T22:02:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.152+0000 7ff3f21b4700 1 -- 172.21.15.173:0/1090116452 learned_addr learned my addr 172.21.15.173:0/1090116452 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.153+0000 7ff3f21b4700 1 -- 172.21.15.173:0/1090116452 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc005040 con 0x7ff3ec0f6510 2024-02-19T22:02:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.153+0000 7ff3f21b4700 1 --2- 172.21.15.173:0/1090116452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec0f6510 0x7ff3ec1089c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7ff3dc005010 tx=0x7ff3dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.153+0000 7ff3e2ffd700 1 -- 172.21.15.173:0/1090116452 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3dc009320 con 0x7ff3ec0f6510 2024-02-19T22:02:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.153+0000 7ff3f4418700 1 -- 172.21.15.173:0/1090116452 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec1090b0 con 0x7ff3ec0f6510 2024-02-19T22:02:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.153+0000 7ff3e2ffd700 1 -- 172.21.15.173:0/1090116452 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3dc013380 con 0x7ff3ec0f6510 2024-02-19T22:02:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.153+0000 7ff3e2ffd700 1 -- 172.21.15.173:0/1090116452 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3dc014e50 con 0x7ff3ec0f6510 2024-02-19T22:02:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.153+0000 7ff3f4418700 1 -- 172.21.15.173:0/1090116452 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec109540 con 0x7ff3ec0f6510 2024-02-19T22:02:30.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.154+0000 7ff3e2ffd700 1 -- 172.21.15.173:0/1090116452 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3dc0095f0 con 0x7ff3ec0f6510 2024-02-19T22:02:30.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.154+0000 7ff3f4418700 1 -- 172.21.15.173:0/1090116452 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec045ad0 con 0x7ff3ec0f6510 2024-02-19T22:02:30.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.154+0000 7ff3e2ffd700 1 --2- 172.21.15.173:0/1090116452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d802ff10 0x7ff3d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:30.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.154+0000 7ff3e2ffd700 1 -- 172.21.15.173:0/1090116452 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3dc040f80 con 0x7ff3ec0f6510 2024-02-19T22:02:30.155 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.154+0000 7ff3f19b3700 1 --2- 172.21.15.173:0/1090116452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d802ff10 0x7ff3d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:30.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.154+0000 7ff3f19b3700 1 --2- 172.21.15.173:0/1090116452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d802ff10 0x7ff3d80323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ff3e80096a0 tx=0x7ff3e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:30.158 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.159+0000 7ff3e2ffd700 1 -- 172.21.15.173:0/1090116452 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3dc014420 con 0x7ff3ec0f6510 2024-02-19T22:02:30.361 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.362+0000 7ff3f4418700 1 -- 172.21.15.173:0/1090116452 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3ec10a3f0 con 0x7ff3ec0f6510 2024-02-19T22:02:30.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.362+0000 7ff3e2ffd700 1 -- 172.21.15.173:0/1090116452 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff3dc010230 con 0x7ff3ec0f6510 2024-02-19T22:02:30.362 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:30.362 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:30.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.364+0000 7ff3f4418700 1 -- 172.21.15.173:0/1090116452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d802ff10 msgr2=0x7ff3d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:30.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.364+0000 7ff3f4418700 1 --2- 172.21.15.173:0/1090116452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d802ff10 0x7ff3d80323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ff3e80096a0 tx=0x7ff3e8006b40).stop 2024-02-19T22:02:30.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.364+0000 7ff3f4418700 1 -- 172.21.15.173:0/1090116452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec0f6510 msgr2=0x7ff3ec1089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:30.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.364+0000 7ff3f4418700 1 --2- 172.21.15.173:0/1090116452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec0f6510 0x7ff3ec1089c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7ff3dc005010 tx=0x7ff3dc00b040).stop 2024-02-19T22:02:30.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.364+0000 7ff3f4418700 1 -- 172.21.15.173:0/1090116452 shutdown_connections 2024-02-19T22:02:30.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.364+0000 7ff3f4418700 1 --2- 172.21.15.173:0/1090116452 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff3d802ff10 0x7ff3d80323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:30.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.364+0000 7ff3f4418700 1 --2- 172.21.15.173:0/1090116452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec0f6510 0x7ff3ec1089c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:30.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.364+0000 7ff3f4418700 1 -- 172.21.15.173:0/1090116452 >> 172.21.15.173:0/1090116452 conn(0x7ff3ec0f1c70 msgr2=0x7ff3ec0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:30.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.364+0000 7ff3f4418700 1 -- 172.21.15.173:0/1090116452 shutdown_connections 2024-02-19T22:02:30.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:30.364+0000 7ff3f4418700 1 -- 172.21.15.173:0/1090116452 wait complete. 2024-02-19T22:02:30.365 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:28.511641+0000 mgr.smithi143.bissbc (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:30.363718+0000 mon.smithi143 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.173:0/1090116452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:32.136 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:32.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:32.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:30.511879+0000 mgr.smithi143.bissbc (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:34.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.067+0000 7efd2d3c9700 1 -- 172.21.15.173:0/2158650149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd280f5410 msgr2=0x7efd280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:34.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.067+0000 7efd2d3c9700 1 --2- 172.21.15.173:0/2158650149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd280f5410 0x7efd280f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7efd1c009fb0 tx=0x7efd1c009b30).stop 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.067+0000 7efd2d3c9700 1 -- 172.21.15.173:0/2158650149 shutdown_connections 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.067+0000 7efd2d3c9700 1 --2- 172.21.15.173:0/2158650149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd280f5410 0x7efd280f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.067+0000 7efd2d3c9700 1 -- 172.21.15.173:0/2158650149 >> 172.21.15.173:0/2158650149 conn(0x7efd280f0b70 msgr2=0x7efd280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.067+0000 7efd2d3c9700 1 -- 172.21.15.173:0/2158650149 shutdown_connections 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.067+0000 7efd2d3c9700 1 -- 172.21.15.173:0/2158650149 wait complete. 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.067+0000 7efd2d3c9700 1 Processor -- start 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.067+0000 7efd2d3c9700 1 -- start start 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.068+0000 7efd2d3c9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd280f5410 0x7efd281056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.068+0000 7efd2d3c9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd28105bd0 con 0x7efd280f5410 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.068+0000 7efd26ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd280f5410 0x7efd281056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.068+0000 7efd26ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd280f5410 0x7efd281056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:47728/0 (socket says 172.21.15.173:47728) 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.068+0000 7efd26ffd700 1 -- 172.21.15.173:0/3882370302 learned_addr learned my addr 172.21.15.173:0/3882370302 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:34.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.068+0000 7efd26ffd700 1 -- 172.21.15.173:0/3882370302 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1c005040 con 0x7efd280f5410 2024-02-19T22:02:34.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.068+0000 7efd26ffd700 1 --2- 172.21.15.173:0/3882370302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd280f5410 0x7efd281056e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7efd1c0089f0 tx=0x7efd1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:34.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.068+0000 7efd0f7fe700 1 -- 172.21.15.173:0/3882370302 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd1c009320 con 0x7efd280f5410 2024-02-19T22:02:34.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.068+0000 7efd2d3c9700 1 -- 172.21.15.173:0/3882370302 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd28105dd0 con 0x7efd280f5410 2024-02-19T22:02:34.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.069+0000 7efd0f7fe700 1 -- 172.21.15.173:0/3882370302 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd1c013380 con 0x7efd280f5410 2024-02-19T22:02:34.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.069+0000 7efd0f7fe700 1 -- 172.21.15.173:0/3882370302 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd1c01ed80 con 0x7efd280f5410 2024-02-19T22:02:34.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.069+0000 7efd2d3c9700 1 -- 172.21.15.173:0/3882370302 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd28106260 con 0x7efd280f5410 2024-02-19T22:02:34.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.069+0000 7efd0f7fe700 1 -- 172.21.15.173:0/3882370302 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd1c014420 con 0x7efd280f5410 2024-02-19T22:02:34.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.069+0000 7efd0f7fe700 1 --2- 172.21.15.173:0/3882370302 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd1002ff60 0x7efd10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:34.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.069+0000 7efd0f7fe700 1 -- 172.21.15.173:0/3882370302 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd1c041050 con 0x7efd280f5410 2024-02-19T22:02:34.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.070+0000 7efd2d3c9700 1 -- 172.21.15.173:0/3882370302 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd28045ad0 con 0x7efd280f5410 2024-02-19T22:02:34.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.070+0000 7efd267fc700 1 --2- 172.21.15.173:0/3882370302 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd1002ff60 0x7efd10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:34.071 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.070+0000 7efd267fc700 1 --2- 172.21.15.173:0/3882370302 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd1002ff60 0x7efd10032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7efd180096a0 tx=0x7efd18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:34.074 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.075+0000 7efd0f7fe700 1 -- 172.21.15.173:0/3882370302 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd1c01c910 con 0x7efd280f5410 2024-02-19T22:02:34.280 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.280+0000 7efd2d3c9700 1 -- 172.21.15.173:0/3882370302 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd28106fb0 con 0x7efd280f5410 2024-02-19T22:02:34.281 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.281+0000 7efd0f7fe700 1 -- 172.21.15.173:0/3882370302 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efd1c01c320 con 0x7efd280f5410 2024-02-19T22:02:34.281 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:34.281 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:34.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.283+0000 7efd2d3c9700 1 -- 172.21.15.173:0/3882370302 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd1002ff60 msgr2=0x7efd10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:34.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.283+0000 7efd2d3c9700 1 --2- 172.21.15.173:0/3882370302 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd1002ff60 0x7efd10032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7efd180096a0 tx=0x7efd18006b40).stop 2024-02-19T22:02:34.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.283+0000 7efd2d3c9700 1 -- 172.21.15.173:0/3882370302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd280f5410 msgr2=0x7efd281056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:34.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.283+0000 7efd2d3c9700 1 --2- 172.21.15.173:0/3882370302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd280f5410 0x7efd281056e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7efd1c0089f0 tx=0x7efd1c00b040).stop 2024-02-19T22:02:34.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.283+0000 7efd2d3c9700 1 -- 172.21.15.173:0/3882370302 shutdown_connections 2024-02-19T22:02:34.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.283+0000 7efd2d3c9700 1 --2- 172.21.15.173:0/3882370302 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7efd1002ff60 0x7efd10032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:34.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.283+0000 7efd2d3c9700 1 --2- 172.21.15.173:0/3882370302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd280f5410 0x7efd281056e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:34.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.283+0000 7efd2d3c9700 1 -- 172.21.15.173:0/3882370302 >> 172.21.15.173:0/3882370302 conn(0x7efd280f0b70 msgr2=0x7efd280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:34.284 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.283+0000 7efd2d3c9700 1 -- 172.21.15.173:0/3882370302 shutdown_connections 2024-02-19T22:02:34.284 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:34.283+0000 7efd2d3c9700 1 -- 172.21.15.173:0/3882370302 wait complete. 2024-02-19T22:02:34.284 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:32.512157+0000 mgr.smithi143.bissbc (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:34.282305+0000 mon.smithi143 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.173:0/3882370302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:36.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:36.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:34.512457+0000 mgr.smithi143.bissbc (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:38.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.107+0000 7f39be945700 1 -- 172.21.15.173:0/1439568517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b80f6510 msgr2=0x7f39b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:38.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.107+0000 7f39be945700 1 --2- 172.21.15.173:0/1439568517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b80f6510 0x7f39b80f6930 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f39ac009fb0 tx=0x7f39ac009b30).stop 2024-02-19T22:02:38.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.107+0000 7f39be945700 1 -- 172.21.15.173:0/1439568517 shutdown_connections 2024-02-19T22:02:38.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.107+0000 7f39be945700 1 --2- 172.21.15.173:0/1439568517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b80f6510 0x7f39b80f6930 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:38.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.107+0000 7f39be945700 1 -- 172.21.15.173:0/1439568517 >> 172.21.15.173:0/1439568517 conn(0x7f39b80f1c70 msgr2=0x7f39b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:38.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.107+0000 7f39be945700 1 -- 172.21.15.173:0/1439568517 shutdown_connections 2024-02-19T22:02:38.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.107+0000 7f39be945700 1 -- 172.21.15.173:0/1439568517 wait complete. 2024-02-19T22:02:38.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.108+0000 7f39be945700 1 Processor -- start 2024-02-19T22:02:38.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.108+0000 7f39be945700 1 -- start start 2024-02-19T22:02:38.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.108+0000 7f39be945700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b80f6510 0x7f39b81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:38.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.108+0000 7f39be945700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8106cd0 con 0x7f39b80f6510 2024-02-19T22:02:38.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.108+0000 7f39b7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b80f6510 0x7f39b81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:38.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.108+0000 7f39b7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b80f6510 0x7f39b81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:47752/0 (socket says 172.21.15.173:47752) 2024-02-19T22:02:38.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.108+0000 7f39b7fff700 1 -- 172.21.15.173:0/1761739574 learned_addr learned my addr 172.21.15.173:0/1761739574 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.109+0000 7f39b7fff700 1 -- 172.21.15.173:0/1761739574 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac005040 con 0x7f39b80f6510 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.109+0000 7f39b7fff700 1 --2- 172.21.15.173:0/1761739574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b80f6510 0x7f39b81067e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f39ac005010 tx=0x7f39ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.109+0000 7f39b4ff9700 1 -- 172.21.15.173:0/1761739574 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39ac009320 con 0x7f39b80f6510 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.109+0000 7f39be945700 1 -- 172.21.15.173:0/1761739574 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b8106ed0 con 0x7f39b80f6510 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.109+0000 7f39b4ff9700 1 -- 172.21.15.173:0/1761739574 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39ac013380 con 0x7f39b80f6510 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.109+0000 7f39b4ff9700 1 -- 172.21.15.173:0/1761739574 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39ac014e50 con 0x7f39b80f6510 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.109+0000 7f39be945700 1 -- 172.21.15.173:0/1761739574 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b8107360 con 0x7f39b80f6510 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.110+0000 7f39b4ff9700 1 -- 172.21.15.173:0/1761739574 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39ac014420 con 0x7f39b80f6510 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.110+0000 7f39be945700 1 -- 172.21.15.173:0/1761739574 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b8045ad0 con 0x7f39b80f6510 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.110+0000 7f39b4ff9700 1 --2- 172.21.15.173:0/1761739574 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f39a002ff10 0x7f39a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.110+0000 7f39b4ff9700 1 -- 172.21.15.173:0/1761739574 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39ac040f40 con 0x7f39b80f6510 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.110+0000 7f39b77fe700 1 --2- 172.21.15.173:0/1761739574 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f39a002ff10 0x7f39a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:38.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.111+0000 7f39b77fe700 1 --2- 172.21.15.173:0/1761739574 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f39a002ff10 0x7f39a00323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f39a80096a0 tx=0x7f39a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:38.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.115+0000 7f39b4ff9700 1 -- 172.21.15.173:0/1761739574 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39ac0095f0 con 0x7f39b80f6510 2024-02-19T22:02:38.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.324+0000 7f39be945700 1 -- 172.21.15.173:0/1761739574 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39b8108210 con 0x7f39b80f6510 2024-02-19T22:02:38.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.325+0000 7f39b4ff9700 1 -- 172.21.15.173:0/1761739574 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f39ac01db70 con 0x7f39b80f6510 2024-02-19T22:02:38.324 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:38.325 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:38.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.326+0000 7f39be945700 1 -- 172.21.15.173:0/1761739574 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f39a002ff10 msgr2=0x7f39a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:38.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.326+0000 7f39be945700 1 --2- 172.21.15.173:0/1761739574 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f39a002ff10 0x7f39a00323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f39a80096a0 tx=0x7f39a8006b40).stop 2024-02-19T22:02:38.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.326+0000 7f39be945700 1 -- 172.21.15.173:0/1761739574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b80f6510 msgr2=0x7f39b81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:38.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.326+0000 7f39be945700 1 --2- 172.21.15.173:0/1761739574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b80f6510 0x7f39b81067e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f39ac005010 tx=0x7f39ac00b040).stop 2024-02-19T22:02:38.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.326+0000 7f39be945700 1 -- 172.21.15.173:0/1761739574 shutdown_connections 2024-02-19T22:02:38.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.326+0000 7f39be945700 1 --2- 172.21.15.173:0/1761739574 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f39a002ff10 0x7f39a00323d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:38.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.326+0000 7f39be945700 1 --2- 172.21.15.173:0/1761739574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b80f6510 0x7f39b81067e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:38.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.326+0000 7f39be945700 1 -- 172.21.15.173:0/1761739574 >> 172.21.15.173:0/1761739574 conn(0x7f39b80f1c70 msgr2=0x7f39b80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:38.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.326+0000 7f39be945700 1 -- 172.21.15.173:0/1761739574 shutdown_connections 2024-02-19T22:02:38.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:38.326+0000 7f39be945700 1 -- 172.21.15.173:0/1761739574 wait complete. 2024-02-19T22:02:38.327 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:36.512798+0000 mgr.smithi143.bissbc (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:38.325810+0000 mon.smithi143 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.173:0/1761739574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:40.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:40.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:38.513134+0000 mgr.smithi143.bissbc (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:42.201 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.201+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/536637008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e40f47c0 msgr2=0x7ff7e40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.201+0000 7ff7ec7d8700 1 --2- 172.21.15.173:0/536637008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e40f47c0 0x7ff7e40f4be0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7ff7e0009fb0 tx=0x7ff7e0009b30).stop 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.201+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/536637008 shutdown_connections 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.201+0000 7ff7ec7d8700 1 --2- 172.21.15.173:0/536637008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e40f47c0 0x7ff7e40f4be0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.201+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/536637008 >> 172.21.15.173:0/536637008 conn(0x7ff7e40f02e0 msgr2=0x7ff7e40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.201+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/536637008 shutdown_connections 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.201+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/536637008 wait complete. 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.201+0000 7ff7ec7d8700 1 Processor -- start 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.201+0000 7ff7ec7d8700 1 -- start start 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.202+0000 7ff7ec7d8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e40f47c0 0x7ff7e41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.202+0000 7ff7ec7d8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e4106cc0 con 0x7ff7e40f47c0 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.202+0000 7ff7ea574700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e40f47c0 0x7ff7e41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:42.202 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.202+0000 7ff7ea574700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e40f47c0 0x7ff7e41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:46930/0 (socket says 172.21.15.173:46930) 2024-02-19T22:02:42.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.202+0000 7ff7ea574700 1 -- 172.21.15.173:0/3909296376 learned_addr learned my addr 172.21.15.173:0/3909296376 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:42.203 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.202+0000 7ff7ea574700 1 -- 172.21.15.173:0/3909296376 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e0005040 con 0x7ff7e40f47c0 2024-02-19T22:02:42.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.202+0000 7ff7ea574700 1 --2- 172.21.15.173:0/3909296376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e40f47c0 0x7ff7e41067d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7ff7e0005010 tx=0x7ff7e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:42.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.203+0000 7ff7daffd700 1 -- 172.21.15.173:0/3909296376 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7e0009320 con 0x7ff7e40f47c0 2024-02-19T22:02:42.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.203+0000 7ff7daffd700 1 -- 172.21.15.173:0/3909296376 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7e0013380 con 0x7ff7e40f47c0 2024-02-19T22:02:42.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.203+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/3909296376 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e4106ec0 con 0x7ff7e40f47c0 2024-02-19T22:02:42.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.203+0000 7ff7daffd700 1 -- 172.21.15.173:0/3909296376 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7e0014e50 con 0x7ff7e40f47c0 2024-02-19T22:02:42.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.203+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/3909296376 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e4107350 con 0x7ff7e40f47c0 2024-02-19T22:02:42.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.203+0000 7ff7daffd700 1 -- 172.21.15.173:0/3909296376 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7e00263a0 con 0x7ff7e40f47c0 2024-02-19T22:02:42.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.204+0000 7ff7daffd700 1 --2- 172.21.15.173:0/3909296376 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7d002ff10 0x7ff7d00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:42.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.204+0000 7ff7daffd700 1 -- 172.21.15.173:0/3909296376 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7e0040f60 con 0x7ff7e40f47c0 2024-02-19T22:02:42.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.204+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/3909296376 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e4045ad0 con 0x7ff7e40f47c0 2024-02-19T22:02:42.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.204+0000 7ff7e9d73700 1 --2- 172.21.15.173:0/3909296376 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7d002ff10 0x7ff7d00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:42.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.204+0000 7ff7e9d73700 1 --2- 172.21.15.173:0/3909296376 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7d002ff10 0x7ff7d00323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7ff7d40096a0 tx=0x7ff7d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:42.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.209+0000 7ff7daffd700 1 -- 172.21.15.173:0/3909296376 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7e0012920 con 0x7ff7e40f47c0 2024-02-19T22:02:42.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.415+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/3909296376 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7e4108200 con 0x7ff7e40f47c0 2024-02-19T22:02:42.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.415+0000 7ff7daffd700 1 -- 172.21.15.173:0/3909296376 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff7e000f340 con 0x7ff7e40f47c0 2024-02-19T22:02:42.415 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:42.415 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:42.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.417+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/3909296376 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7d002ff10 msgr2=0x7ff7d00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:42.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.417+0000 7ff7ec7d8700 1 --2- 172.21.15.173:0/3909296376 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7d002ff10 0x7ff7d00323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7ff7d40096a0 tx=0x7ff7d4006b40).stop 2024-02-19T22:02:42.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.417+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/3909296376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e40f47c0 msgr2=0x7ff7e41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:42.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.417+0000 7ff7ec7d8700 1 --2- 172.21.15.173:0/3909296376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e40f47c0 0x7ff7e41067d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7ff7e0005010 tx=0x7ff7e000b040).stop 2024-02-19T22:02:42.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.417+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/3909296376 shutdown_connections 2024-02-19T22:02:42.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.417+0000 7ff7ec7d8700 1 --2- 172.21.15.173:0/3909296376 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7d002ff10 0x7ff7d00323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:42.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.417+0000 7ff7ec7d8700 1 --2- 172.21.15.173:0/3909296376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e40f47c0 0x7ff7e41067d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:42.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.417+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/3909296376 >> 172.21.15.173:0/3909296376 conn(0x7ff7e40f02e0 msgr2=0x7ff7e40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:42.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.417+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/3909296376 shutdown_connections 2024-02-19T22:02:42.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:42.417+0000 7ff7ec7d8700 1 -- 172.21.15.173:0/3909296376 wait complete. 2024-02-19T22:02:42.417 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:40.513560+0000 mgr.smithi143.bissbc (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:42.416504+0000 mon.smithi143 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.173:0/3909296376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:44.243 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:44.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:42.513924+0000 mgr.smithi143.bissbc (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:46.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.279+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/2657200207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f80f5410 msgr2=0x7fd4f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:46.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.279+0000 7fd4ff7a8700 1 --2- 172.21.15.173:0/2657200207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f80f5410 0x7fd4f80f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fd4f4009fb0 tx=0x7fd4f4009b30).stop 2024-02-19T22:02:46.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.279+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/2657200207 shutdown_connections 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.279+0000 7fd4ff7a8700 1 --2- 172.21.15.173:0/2657200207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f80f5410 0x7fd4f80f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.279+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/2657200207 >> 172.21.15.173:0/2657200207 conn(0x7fd4f80f0b70 msgr2=0x7fd4f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.279+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/2657200207 shutdown_connections 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.279+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/2657200207 wait complete. 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.280+0000 7fd4ff7a8700 1 Processor -- start 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.280+0000 7fd4ff7a8700 1 -- start start 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.280+0000 7fd4ff7a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f80f5410 0x7fd4f810b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.280+0000 7fd4ff7a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f8107330 con 0x7fd4f80f5410 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.280+0000 7fd4fd544700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f80f5410 0x7fd4f810b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.280+0000 7fd4fd544700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f80f5410 0x7fd4f810b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:46962/0 (socket says 172.21.15.173:46962) 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.280+0000 7fd4fd544700 1 -- 172.21.15.173:0/505271640 learned_addr learned my addr 172.21.15.173:0/505271640 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.280+0000 7fd4fd544700 1 -- 172.21.15.173:0/505271640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f4005040 con 0x7fd4f80f5410 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.280+0000 7fd4fd544700 1 --2- 172.21.15.173:0/505271640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f80f5410 0x7fd4f810b940 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fd4f4005010 tx=0x7fd4f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:46.323 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.280+0000 7fd4edffb700 1 -- 172.21.15.173:0/505271640 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4f4009320 con 0x7fd4f80f5410 2024-02-19T22:02:46.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.281+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/505271640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f8060220 con 0x7fd4f80f5410 2024-02-19T22:02:46.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.281+0000 7fd4edffb700 1 -- 172.21.15.173:0/505271640 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4f4013380 con 0x7fd4f80f5410 2024-02-19T22:02:46.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.281+0000 7fd4edffb700 1 -- 172.21.15.173:0/505271640 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4f4014e50 con 0x7fd4f80f5410 2024-02-19T22:02:46.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.281+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/505271640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f810be30 con 0x7fd4f80f5410 2024-02-19T22:02:46.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.282+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/505271640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f8045ad0 con 0x7fd4f80f5410 2024-02-19T22:02:46.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.282+0000 7fd4edffb700 1 -- 172.21.15.173:0/505271640 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4f40095f0 con 0x7fd4f80f5410 2024-02-19T22:02:46.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.282+0000 7fd4edffb700 1 --2- 172.21.15.173:0/505271640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd4e402ff10 0x7fd4e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:46.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.282+0000 7fd4edffb700 1 -- 172.21.15.173:0/505271640 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4f4040fa0 con 0x7fd4f80f5410 2024-02-19T22:02:46.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.282+0000 7fd4fcd43700 1 --2- 172.21.15.173:0/505271640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd4e402ff10 0x7fd4e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:46.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.283+0000 7fd4fcd43700 1 --2- 172.21.15.173:0/505271640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd4e402ff10 0x7fd4e40323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fd4e80096a0 tx=0x7fd4e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:46.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.287+0000 7fd4edffb700 1 -- 172.21.15.173:0/505271640 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4f4014420 con 0x7fd4f80f5410 2024-02-19T22:02:46.496 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.496+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/505271640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4f810c5f0 con 0x7fd4f80f5410 2024-02-19T22:02:46.496 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.497+0000 7fd4edffb700 1 -- 172.21.15.173:0/505271640 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd4f4010230 con 0x7fd4f80f5410 2024-02-19T22:02:46.496 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:46.497 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:46.498 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.498+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/505271640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd4e402ff10 msgr2=0x7fd4e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:46.498 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.498+0000 7fd4ff7a8700 1 --2- 172.21.15.173:0/505271640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd4e402ff10 0x7fd4e40323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fd4e80096a0 tx=0x7fd4e8006b40).stop 2024-02-19T22:02:46.498 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.498+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/505271640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f80f5410 msgr2=0x7fd4f810b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:46.498 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.498+0000 7fd4ff7a8700 1 --2- 172.21.15.173:0/505271640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f80f5410 0x7fd4f810b940 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fd4f4005010 tx=0x7fd4f400b040).stop 2024-02-19T22:02:46.498 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.498+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/505271640 shutdown_connections 2024-02-19T22:02:46.498 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.498+0000 7fd4ff7a8700 1 --2- 172.21.15.173:0/505271640 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd4e402ff10 0x7fd4e40323d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:46.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.498+0000 7fd4ff7a8700 1 --2- 172.21.15.173:0/505271640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f80f5410 0x7fd4f810b940 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:46.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.498+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/505271640 >> 172.21.15.173:0/505271640 conn(0x7fd4f80f0b70 msgr2=0x7fd4f80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:46.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.498+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/505271640 shutdown_connections 2024-02-19T22:02:46.499 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:46.498+0000 7fd4ff7a8700 1 -- 172.21.15.173:0/505271640 wait complete. 2024-02-19T22:02:46.499 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:44.514201+0000 mgr.smithi143.bissbc (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:46.497930+0000 mon.smithi143 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.173:0/505271640' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:48.205 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:48.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:46.514491+0000 mgr.smithi143.bissbc (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:50.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.160+0000 7ffa002f0700 1 -- 172.21.15.173:0/2495170390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9f80f5410 msgr2=0x7ff9f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:50.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.160+0000 7ffa002f0700 1 --2- 172.21.15.173:0/2495170390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9f80f5410 0x7ff9f80f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7ff9ec009fb0 tx=0x7ff9ec009b30).stop 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.160+0000 7ffa002f0700 1 -- 172.21.15.173:0/2495170390 shutdown_connections 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.160+0000 7ffa002f0700 1 --2- 172.21.15.173:0/2495170390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9f80f5410 0x7ff9f80f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.160+0000 7ffa002f0700 1 -- 172.21.15.173:0/2495170390 >> 172.21.15.173:0/2495170390 conn(0x7ff9f80f0b70 msgr2=0x7ff9f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.160+0000 7ffa002f0700 1 -- 172.21.15.173:0/2495170390 shutdown_connections 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.160+0000 7ffa002f0700 1 -- 172.21.15.173:0/2495170390 wait complete. 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.161+0000 7ffa002f0700 1 Processor -- start 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.161+0000 7ffa002f0700 1 -- start start 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.161+0000 7ffa002f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9f80f5410 0x7ff9f81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.161+0000 7ffa002f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f8105bd0 con 0x7ff9f80f5410 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.162+0000 7ff9fe08c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9f80f5410 0x7ff9f81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.162+0000 7ff9fe08c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9f80f5410 0x7ff9f81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:46972/0 (socket says 172.21.15.173:46972) 2024-02-19T22:02:50.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.162+0000 7ff9fe08c700 1 -- 172.21.15.173:0/382723381 learned_addr learned my addr 172.21.15.173:0/382723381 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:50.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.162+0000 7ff9fe08c700 1 -- 172.21.15.173:0/382723381 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec005040 con 0x7ff9f80f5410 2024-02-19T22:02:50.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.162+0000 7ff9fe08c700 1 --2- 172.21.15.173:0/382723381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9f80f5410 0x7ff9f81056e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7ff9ec005010 tx=0x7ff9ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:50.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.162+0000 7ff9eaffd700 1 -- 172.21.15.173:0/382723381 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9ec009320 con 0x7ff9f80f5410 2024-02-19T22:02:50.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.162+0000 7ffa002f0700 1 -- 172.21.15.173:0/382723381 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f8105dd0 con 0x7ff9f80f5410 2024-02-19T22:02:50.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.162+0000 7ff9eaffd700 1 -- 172.21.15.173:0/382723381 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9ec013380 con 0x7ff9f80f5410 2024-02-19T22:02:50.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.163+0000 7ff9eaffd700 1 -- 172.21.15.173:0/382723381 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9ec014e50 con 0x7ff9f80f5410 2024-02-19T22:02:50.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.163+0000 7ffa002f0700 1 -- 172.21.15.173:0/382723381 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f8106260 con 0x7ff9f80f5410 2024-02-19T22:02:50.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.163+0000 7ff9eaffd700 1 -- 172.21.15.173:0/382723381 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9ec0263a0 con 0x7ff9f80f5410 2024-02-19T22:02:50.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.163+0000 7ffa002f0700 1 -- 172.21.15.173:0/382723381 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f8045ad0 con 0x7ff9f80f5410 2024-02-19T22:02:50.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.163+0000 7ff9eaffd700 1 --2- 172.21.15.173:0/382723381 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9e402ff10 0x7ff9e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:50.168 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.164+0000 7ff9eaffd700 1 -- 172.21.15.173:0/382723381 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9ec0412e0 con 0x7ff9f80f5410 2024-02-19T22:02:50.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.164+0000 7ff9fd88b700 1 --2- 172.21.15.173:0/382723381 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9e402ff10 0x7ff9e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:50.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.164+0000 7ff9fd88b700 1 --2- 172.21.15.173:0/382723381 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9e402ff10 0x7ff9e40323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7ff9f40096a0 tx=0x7ff9f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:50.169 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.169+0000 7ff9eaffd700 1 -- 172.21.15.173:0/382723381 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9ec01c920 con 0x7ff9f80f5410 2024-02-19T22:02:50.372 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.373+0000 7ffa002f0700 1 -- 172.21.15.173:0/382723381 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9f8107110 con 0x7ff9f80f5410 2024-02-19T22:02:50.373 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.374+0000 7ff9eaffd700 1 -- 172.21.15.173:0/382723381 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff9ec01c920 con 0x7ff9f80f5410 2024-02-19T22:02:50.374 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:50.374 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:50.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.375+0000 7ffa002f0700 1 -- 172.21.15.173:0/382723381 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9e402ff10 msgr2=0x7ff9e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:50.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.375+0000 7ffa002f0700 1 --2- 172.21.15.173:0/382723381 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9e402ff10 0x7ff9e40323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7ff9f40096a0 tx=0x7ff9f4006b40).stop 2024-02-19T22:02:50.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.375+0000 7ffa002f0700 1 -- 172.21.15.173:0/382723381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9f80f5410 msgr2=0x7ff9f81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:50.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.375+0000 7ffa002f0700 1 --2- 172.21.15.173:0/382723381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9f80f5410 0x7ff9f81056e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7ff9ec005010 tx=0x7ff9ec00b040).stop 2024-02-19T22:02:50.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.375+0000 7ffa002f0700 1 -- 172.21.15.173:0/382723381 shutdown_connections 2024-02-19T22:02:50.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.375+0000 7ffa002f0700 1 --2- 172.21.15.173:0/382723381 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9e402ff10 0x7ff9e40323d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:50.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.375+0000 7ffa002f0700 1 --2- 172.21.15.173:0/382723381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9f80f5410 0x7ff9f81056e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:50.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.375+0000 7ffa002f0700 1 -- 172.21.15.173:0/382723381 >> 172.21.15.173:0/382723381 conn(0x7ff9f80f0b70 msgr2=0x7ff9f80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:50.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.375+0000 7ffa002f0700 1 -- 172.21.15.173:0/382723381 shutdown_connections 2024-02-19T22:02:50.376 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:50.375+0000 7ffa002f0700 1 -- 172.21.15.173:0/382723381 wait complete. 2024-02-19T22:02:50.376 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:48.514819+0000 mgr.smithi143.bissbc (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:50.374873+0000 mon.smithi143 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.173:0/382723381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:52.041 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:52.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:52.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:50.515047+0000 mgr.smithi143.bissbc (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:53.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.962+0000 7fa7a35a8700 1 -- 172.21.15.173:0/2310406323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa79c0f5410 msgr2=0x7fa79c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.962+0000 7fa7a35a8700 1 --2- 172.21.15.173:0/2310406323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa79c0f5410 0x7fa79c0f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fa78c009fb0 tx=0x7fa78c009b30).stop 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.962+0000 7fa7a35a8700 1 -- 172.21.15.173:0/2310406323 shutdown_connections 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.962+0000 7fa7a35a8700 1 --2- 172.21.15.173:0/2310406323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa79c0f5410 0x7fa79c0f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.962+0000 7fa7a35a8700 1 -- 172.21.15.173:0/2310406323 >> 172.21.15.173:0/2310406323 conn(0x7fa79c0f0b70 msgr2=0x7fa79c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.962+0000 7fa7a35a8700 1 -- 172.21.15.173:0/2310406323 shutdown_connections 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.962+0000 7fa7a35a8700 1 -- 172.21.15.173:0/2310406323 wait complete. 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.963+0000 7fa7a35a8700 1 Processor -- start 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.963+0000 7fa7a35a8700 1 -- start start 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.963+0000 7fa7a35a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa79c0f5410 0x7fa79c109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.963+0000 7fa7a35a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c060220 con 0x7fa79c0f5410 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.963+0000 7fa7a1344700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa79c0f5410 0x7fa79c109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.963+0000 7fa7a1344700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa79c0f5410 0x7fa79c109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41840/0 (socket says 172.21.15.173:41840) 2024-02-19T22:02:53.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.963+0000 7fa7a1344700 1 -- 172.21.15.173:0/1198779221 learned_addr learned my addr 172.21.15.173:0/1198779221 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:53.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.963+0000 7fa7a1344700 1 -- 172.21.15.173:0/1198779221 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c005040 con 0x7fa79c0f5410 2024-02-19T22:02:53.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.963+0000 7fa7a1344700 1 --2- 172.21.15.173:0/1198779221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa79c0f5410 0x7fa79c109540 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fa78c005010 tx=0x7fa78c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:53.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.964+0000 7fa791ffb700 1 -- 172.21.15.173:0/1198779221 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa78c009320 con 0x7fa79c0f5410 2024-02-19T22:02:53.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.964+0000 7fa7a35a8700 1 -- 172.21.15.173:0/1198779221 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79c060420 con 0x7fa79c0f5410 2024-02-19T22:02:53.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.964+0000 7fa791ffb700 1 -- 172.21.15.173:0/1198779221 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa78c013380 con 0x7fa79c0f5410 2024-02-19T22:02:53.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.964+0000 7fa791ffb700 1 -- 172.21.15.173:0/1198779221 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa78c014e50 con 0x7fa79c0f5410 2024-02-19T22:02:53.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.964+0000 7fa7a35a8700 1 -- 172.21.15.173:0/1198779221 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa79c109bf0 con 0x7fa79c0f5410 2024-02-19T22:02:53.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.964+0000 7fa791ffb700 1 -- 172.21.15.173:0/1198779221 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa78c0263a0 con 0x7fa79c0f5410 2024-02-19T22:02:53.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.965+0000 7fa7a35a8700 1 -- 172.21.15.173:0/1198779221 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79c059100 con 0x7fa79c0f5410 2024-02-19T22:02:53.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.965+0000 7fa791ffb700 1 --2- 172.21.15.173:0/1198779221 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa78802ff10 0x7fa7880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:53.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.965+0000 7fa791ffb700 1 -- 172.21.15.173:0/1198779221 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa78c041250 con 0x7fa79c0f5410 2024-02-19T22:02:53.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.965+0000 7fa7a0b43700 1 --2- 172.21.15.173:0/1198779221 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa78802ff10 0x7fa7880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:53.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.965+0000 7fa7a0b43700 1 --2- 172.21.15.173:0/1198779221 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa78802ff10 0x7fa7880323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa7980096a0 tx=0x7fa798006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:53.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:53.970+0000 7fa791ffb700 1 -- 172.21.15.173:0/1198779221 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa78c01cca0 con 0x7fa79c0f5410 2024-02-19T22:02:54.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.176+0000 7fa7a35a8700 1 -- 172.21.15.173:0/1198779221 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa79c045ad0 con 0x7fa79c0f5410 2024-02-19T22:02:54.176 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.177+0000 7fa791ffb700 1 -- 172.21.15.173:0/1198779221 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa78c012330 con 0x7fa79c0f5410 2024-02-19T22:02:54.177 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:54.177 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:54.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.178+0000 7fa7a35a8700 1 -- 172.21.15.173:0/1198779221 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa78802ff10 msgr2=0x7fa7880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:54.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.178+0000 7fa7a35a8700 1 --2- 172.21.15.173:0/1198779221 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa78802ff10 0x7fa7880323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa7980096a0 tx=0x7fa798006b40).stop 2024-02-19T22:02:54.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.178+0000 7fa7a35a8700 1 -- 172.21.15.173:0/1198779221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa79c0f5410 msgr2=0x7fa79c109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:54.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.178+0000 7fa7a35a8700 1 --2- 172.21.15.173:0/1198779221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa79c0f5410 0x7fa79c109540 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fa78c005010 tx=0x7fa78c00b040).stop 2024-02-19T22:02:54.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.179+0000 7fa7a35a8700 1 -- 172.21.15.173:0/1198779221 shutdown_connections 2024-02-19T22:02:54.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.179+0000 7fa7a35a8700 1 --2- 172.21.15.173:0/1198779221 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa78802ff10 0x7fa7880323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:54.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.179+0000 7fa7a35a8700 1 --2- 172.21.15.173:0/1198779221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa79c0f5410 0x7fa79c109540 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:54.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.179+0000 7fa7a35a8700 1 -- 172.21.15.173:0/1198779221 >> 172.21.15.173:0/1198779221 conn(0x7fa79c0f0b70 msgr2=0x7fa79c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:54.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.179+0000 7fa7a35a8700 1 -- 172.21.15.173:0/1198779221 shutdown_connections 2024-02-19T22:02:54.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:54.179+0000 7fa7a35a8700 1 -- 172.21.15.173:0/1198779221 wait complete. 2024-02-19T22:02:54.179 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:52.515288+0000 mgr.smithi143.bissbc (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:54.178160+0000 mon.smithi143 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.173:0/1198779221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:55.869 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:55.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:55.909 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:55 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:54.515515+0000 mgr.smithi143.bissbc (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:57.918 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.917+0000 7f99d9a84700 1 -- 172.21.15.173:0/2669874375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d40f5410 msgr2=0x7f99d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.917+0000 7f99d9a84700 1 --2- 172.21.15.173:0/2669874375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d40f5410 0x7f99d40f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f99c8009fb0 tx=0x7f99c8009b30).stop 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.918+0000 7f99d9a84700 1 -- 172.21.15.173:0/2669874375 shutdown_connections 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.918+0000 7f99d9a84700 1 --2- 172.21.15.173:0/2669874375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d40f5410 0x7f99d40f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.918+0000 7f99d9a84700 1 -- 172.21.15.173:0/2669874375 >> 172.21.15.173:0/2669874375 conn(0x7f99d40f0b70 msgr2=0x7f99d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.918+0000 7f99d9a84700 1 -- 172.21.15.173:0/2669874375 shutdown_connections 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.918+0000 7f99d9a84700 1 -- 172.21.15.173:0/2669874375 wait complete. 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.918+0000 7f99d9a84700 1 Processor -- start 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.918+0000 7f99d9a84700 1 -- start start 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.918+0000 7f99d9a84700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d40f5410 0x7f99d41089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.918+0000 7f99d9a84700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d4108ed0 con 0x7f99d40f5410 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.919+0000 7f99d37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d40f5410 0x7f99d41089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.919+0000 7f99d37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d40f5410 0x7f99d41089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41850/0 (socket says 172.21.15.173:41850) 2024-02-19T22:02:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.919+0000 7f99d37fe700 1 -- 172.21.15.173:0/3883737138 learned_addr learned my addr 172.21.15.173:0/3883737138 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:02:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.919+0000 7f99d37fe700 1 -- 172.21.15.173:0/3883737138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c8005040 con 0x7f99d40f5410 2024-02-19T22:02:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.919+0000 7f99d37fe700 1 --2- 172.21.15.173:0/3883737138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d40f5410 0x7f99d41089e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f99c8005010 tx=0x7f99c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.919+0000 7f99bbfff700 1 -- 172.21.15.173:0/3883737138 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99c8009320 con 0x7f99d40f5410 2024-02-19T22:02:57.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.919+0000 7f99d9a84700 1 -- 172.21.15.173:0/3883737138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d41090d0 con 0x7f99d40f5410 2024-02-19T22:02:57.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.919+0000 7f99bbfff700 1 -- 172.21.15.173:0/3883737138 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99c8013380 con 0x7f99d40f5410 2024-02-19T22:02:57.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.919+0000 7f99bbfff700 1 -- 172.21.15.173:0/3883737138 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99c8014e50 con 0x7f99d40f5410 2024-02-19T22:02:57.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.919+0000 7f99d9a84700 1 -- 172.21.15.173:0/3883737138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d4109560 con 0x7f99d40f5410 2024-02-19T22:02:57.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.920+0000 7f99bbfff700 1 -- 172.21.15.173:0/3883737138 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99c8014420 con 0x7f99d40f5410 2024-02-19T22:02:57.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.920+0000 7f99d9a84700 1 -- 172.21.15.173:0/3883737138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d4045ad0 con 0x7f99d40f5410 2024-02-19T22:02:57.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.920+0000 7f99bbfff700 1 --2- 172.21.15.173:0/3883737138 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f99bc02ff10 0x7f99bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:57.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.920+0000 7f99bbfff700 1 -- 172.21.15.173:0/3883737138 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99c8040f40 con 0x7f99d40f5410 2024-02-19T22:02:57.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.920+0000 7f99d2ffd700 1 --2- 172.21.15.173:0/3883737138 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f99bc02ff10 0x7f99bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:57.921 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.921+0000 7f99d2ffd700 1 --2- 172.21.15.173:0/3883737138 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f99bc02ff10 0x7f99bc0323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f99c40096a0 tx=0x7f99c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:57.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:57.925+0000 7f99bbfff700 1 -- 172.21.15.173:0/3883737138 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99c80095f0 con 0x7f99d40f5410 2024-02-19T22:02:58.128 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.129+0000 7f99d9a84700 1 -- 172.21.15.173:0/3883737138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99d410a410 con 0x7f99d40f5410 2024-02-19T22:02:58.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.129+0000 7f99bbfff700 1 -- 172.21.15.173:0/3883737138 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f99c801db70 con 0x7f99d40f5410 2024-02-19T22:02:58.129 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:02:58.130 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:58.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.131+0000 7f99d9a84700 1 -- 172.21.15.173:0/3883737138 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f99bc02ff10 msgr2=0x7f99bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:58.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.131+0000 7f99d9a84700 1 --2- 172.21.15.173:0/3883737138 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f99bc02ff10 0x7f99bc0323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f99c40096a0 tx=0x7f99c4006b40).stop 2024-02-19T22:02:58.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.131+0000 7f99d9a84700 1 -- 172.21.15.173:0/3883737138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d40f5410 msgr2=0x7f99d41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:58.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.131+0000 7f99d9a84700 1 --2- 172.21.15.173:0/3883737138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d40f5410 0x7f99d41089e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f99c8005010 tx=0x7f99c800b040).stop 2024-02-19T22:02:58.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.131+0000 7f99d9a84700 1 -- 172.21.15.173:0/3883737138 shutdown_connections 2024-02-19T22:02:58.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.131+0000 7f99d9a84700 1 --2- 172.21.15.173:0/3883737138 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f99bc02ff10 0x7f99bc0323d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:58.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.131+0000 7f99d9a84700 1 --2- 172.21.15.173:0/3883737138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d40f5410 0x7f99d41089e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:58.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.131+0000 7f99d9a84700 1 -- 172.21.15.173:0/3883737138 >> 172.21.15.173:0/3883737138 conn(0x7f99d40f0b70 msgr2=0x7f99d40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:58.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.131+0000 7f99d9a84700 1 -- 172.21.15.173:0/3883737138 shutdown_connections 2024-02-19T22:02:58.131 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:02:58.131+0000 7f99d9a84700 1 -- 172.21.15.173:0/3883737138 wait complete. 2024-02-19T22:02:58.132 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:02:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:56.515853+0000 mgr.smithi143.bissbc (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:02:58.130839+0000 mon.smithi143 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.173:0/3883737138' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:59.881 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:59.881 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:59.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:02:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:02:58.516136+0000 mgr.smithi143.bissbc (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:01.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.746+0000 7f1b07fff700 1 -- 172.21.15.173:0/3556858468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b080f62a0 msgr2=0x7f1b080f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:01.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.746+0000 7f1b07fff700 1 --2- 172.21.15.173:0/3556858468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b080f62a0 0x7f1b080f66c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f1af8009fb0 tx=0x7f1af8009b30).stop 2024-02-19T22:03:01.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.746+0000 7f1b07fff700 1 -- 172.21.15.173:0/3556858468 shutdown_connections 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.746+0000 7f1b07fff700 1 --2- 172.21.15.173:0/3556858468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b080f62a0 0x7f1b080f66c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.747+0000 7f1b07fff700 1 -- 172.21.15.173:0/3556858468 >> 172.21.15.173:0/3556858468 conn(0x7f1b080f1aa0 msgr2=0x7f1b080f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.747+0000 7f1b07fff700 1 -- 172.21.15.173:0/3556858468 shutdown_connections 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.747+0000 7f1b07fff700 1 -- 172.21.15.173:0/3556858468 wait complete. 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.747+0000 7f1b07fff700 1 Processor -- start 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.747+0000 7f1b07fff700 1 -- start start 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.748+0000 7f1b07fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b080f62a0 0x7f1b081065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.748+0000 7f1b07fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b08106ac0 con 0x7f1b080f62a0 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.748+0000 7f1b06ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b080f62a0 0x7f1b081065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.748+0000 7f1b06ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b080f62a0 0x7f1b081065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41476/0 (socket says 172.21.15.173:41476) 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.748+0000 7f1b06ffd700 1 -- 172.21.15.173:0/1439736831 learned_addr learned my addr 172.21.15.173:0/1439736831 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.748+0000 7f1b06ffd700 1 -- 172.21.15.173:0/1439736831 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af8005040 con 0x7f1b080f62a0 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.748+0000 7f1b06ffd700 1 --2- 172.21.15.173:0/1439736831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b080f62a0 0x7f1b081065d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f1af80089f0 tx=0x7f1af800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:01.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.748+0000 7f1aef7fe700 1 -- 172.21.15.173:0/1439736831 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af8009320 con 0x7f1b080f62a0 2024-02-19T22:03:01.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.749+0000 7f1b07fff700 1 -- 172.21.15.173:0/1439736831 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b08106cc0 con 0x7f1b080f62a0 2024-02-19T22:03:01.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.749+0000 7f1aef7fe700 1 -- 172.21.15.173:0/1439736831 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1af8013380 con 0x7f1b080f62a0 2024-02-19T22:03:01.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.749+0000 7f1aef7fe700 1 -- 172.21.15.173:0/1439736831 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af8014e20 con 0x7f1b080f62a0 2024-02-19T22:03:01.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.749+0000 7f1b07fff700 1 -- 172.21.15.173:0/1439736831 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b08107150 con 0x7f1b080f62a0 2024-02-19T22:03:01.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.749+0000 7f1aef7fe700 1 -- 172.21.15.173:0/1439736831 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1af8014420 con 0x7f1b080f62a0 2024-02-19T22:03:01.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.749+0000 7f1b07fff700 1 -- 172.21.15.173:0/1439736831 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b08045ad0 con 0x7f1b080f62a0 2024-02-19T22:03:01.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.750+0000 7f1aef7fe700 1 --2- 172.21.15.173:0/1439736831 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1af002ff10 0x7f1af00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:01.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.750+0000 7f1aef7fe700 1 -- 172.21.15.173:0/1439736831 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1af8040f50 con 0x7f1b080f62a0 2024-02-19T22:03:01.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.750+0000 7f1b067fc700 1 --2- 172.21.15.173:0/1439736831 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1af002ff10 0x7f1af00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:01.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.750+0000 7f1b067fc700 1 --2- 172.21.15.173:0/1439736831 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1af002ff10 0x7f1af00323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f1afc0096a0 tx=0x7f1afc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:01.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.755+0000 7f1aef7fe700 1 -- 172.21.15.173:0/1439736831 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1af806d2b0 con 0x7f1b080f62a0 2024-02-19T22:03:01.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:01 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:00.516597+0000 mgr.smithi143.bissbc (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:01.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.958+0000 7f1b07fff700 1 -- 172.21.15.173:0/1439736831 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b08108000 con 0x7f1b080f62a0 2024-02-19T22:03:01.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.959+0000 7f1aef7fe700 1 -- 172.21.15.173:0/1439736831 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1af80095f0 con 0x7f1b080f62a0 2024-02-19T22:03:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:01.959 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:01.961 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.960+0000 7f1b07fff700 1 -- 172.21.15.173:0/1439736831 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1af002ff10 msgr2=0x7f1af00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:01.961 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.960+0000 7f1b07fff700 1 --2- 172.21.15.173:0/1439736831 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1af002ff10 0x7f1af00323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f1afc0096a0 tx=0x7f1afc006b40).stop 2024-02-19T22:03:01.961 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.960+0000 7f1b07fff700 1 -- 172.21.15.173:0/1439736831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b080f62a0 msgr2=0x7f1b081065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:01.961 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.960+0000 7f1b07fff700 1 --2- 172.21.15.173:0/1439736831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b080f62a0 0x7f1b081065d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f1af80089f0 tx=0x7f1af800b040).stop 2024-02-19T22:03:01.961 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.960+0000 7f1b07fff700 1 -- 172.21.15.173:0/1439736831 shutdown_connections 2024-02-19T22:03:01.962 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.960+0000 7f1b07fff700 1 --2- 172.21.15.173:0/1439736831 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1af002ff10 0x7f1af00323d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:01.962 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.960+0000 7f1b07fff700 1 --2- 172.21.15.173:0/1439736831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b080f62a0 0x7f1b081065d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:01.962 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.960+0000 7f1b07fff700 1 -- 172.21.15.173:0/1439736831 >> 172.21.15.173:0/1439736831 conn(0x7f1b080f1aa0 msgr2=0x7f1b080f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:01.962 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.960+0000 7f1b07fff700 1 -- 172.21.15.173:0/1439736831 shutdown_connections 2024-02-19T22:03:01.962 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:01.960+0000 7f1b07fff700 1 -- 172.21.15.173:0/1439736831 wait complete. 2024-02-19T22:03:01.962 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:01.960706+0000 mon.smithi143 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.173:0/1439736831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:03.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:03.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:02.517032+0000 mgr.smithi143.bissbc (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:05.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.821+0000 7f48c2bd9700 1 -- 172.21.15.173:0/1123093110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48bc0f5410 msgr2=0x7f48bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.821+0000 7f48c2bd9700 1 --2- 172.21.15.173:0/1123093110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48bc0f5410 0x7f48bc0f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f48b0009fb0 tx=0x7f48b0009b30).stop 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.821+0000 7f48c2bd9700 1 -- 172.21.15.173:0/1123093110 shutdown_connections 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.821+0000 7f48c2bd9700 1 --2- 172.21.15.173:0/1123093110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48bc0f5410 0x7f48bc0f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.821+0000 7f48c2bd9700 1 -- 172.21.15.173:0/1123093110 >> 172.21.15.173:0/1123093110 conn(0x7f48bc0f0b70 msgr2=0x7f48bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.821+0000 7f48c2bd9700 1 -- 172.21.15.173:0/1123093110 shutdown_connections 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.821+0000 7f48c2bd9700 1 -- 172.21.15.173:0/1123093110 wait complete. 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.822+0000 7f48c2bd9700 1 Processor -- start 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.822+0000 7f48c2bd9700 1 -- start start 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.822+0000 7f48c2bd9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48bc0f5410 0x7f48bc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.822+0000 7f48c2bd9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc105bd0 con 0x7f48bc0f5410 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.823+0000 7f48c0975700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48bc0f5410 0x7f48bc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:05.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.823+0000 7f48c0975700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48bc0f5410 0x7f48bc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41492/0 (socket says 172.21.15.173:41492) 2024-02-19T22:03:05.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.823+0000 7f48c0975700 1 -- 172.21.15.173:0/3657035728 learned_addr learned my addr 172.21.15.173:0/3657035728 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:05.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.823+0000 7f48c0975700 1 -- 172.21.15.173:0/3657035728 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b0005040 con 0x7f48bc0f5410 2024-02-19T22:03:05.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.823+0000 7f48c0975700 1 --2- 172.21.15.173:0/3657035728 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48bc0f5410 0x7f48bc1056e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f48b00089f0 tx=0x7f48b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:05.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.823+0000 7f48b97fa700 1 -- 172.21.15.173:0/3657035728 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b0009320 con 0x7f48bc0f5410 2024-02-19T22:03:05.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.823+0000 7f48c2bd9700 1 -- 172.21.15.173:0/3657035728 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc105dd0 con 0x7f48bc0f5410 2024-02-19T22:03:05.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.823+0000 7f48b97fa700 1 -- 172.21.15.173:0/3657035728 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48b0013380 con 0x7f48bc0f5410 2024-02-19T22:03:05.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.823+0000 7f48b97fa700 1 -- 172.21.15.173:0/3657035728 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b001ed80 con 0x7f48bc0f5410 2024-02-19T22:03:05.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.823+0000 7f48c2bd9700 1 -- 172.21.15.173:0/3657035728 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc106260 con 0x7f48bc0f5410 2024-02-19T22:03:05.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.824+0000 7f48b97fa700 1 -- 172.21.15.173:0/3657035728 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48b0014420 con 0x7f48bc0f5410 2024-02-19T22:03:05.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.824+0000 7f48c2bd9700 1 -- 172.21.15.173:0/3657035728 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc045ad0 con 0x7f48bc0f5410 2024-02-19T22:03:05.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.824+0000 7f48b97fa700 1 --2- 172.21.15.173:0/3657035728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f48ac02ff60 0x7f48ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:05.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.824+0000 7f48b97fa700 1 -- 172.21.15.173:0/3657035728 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48b0041210 con 0x7f48bc0f5410 2024-02-19T22:03:05.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.824+0000 7f48bbfff700 1 --2- 172.21.15.173:0/3657035728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f48ac02ff60 0x7f48ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:05.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.825+0000 7f48bbfff700 1 --2- 172.21.15.173:0/3657035728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f48ac02ff60 0x7f48ac032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f48a80096a0 tx=0x7f48a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:05.828 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:05.829+0000 7f48b97fa700 1 -- 172.21.15.173:0/3657035728 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48b001c500 con 0x7f48bc0f5410 2024-02-19T22:03:06.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.035+0000 7f48c2bd9700 1 -- 172.21.15.173:0/3657035728 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48bc107110 con 0x7f48bc0f5410 2024-02-19T22:03:06.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.036+0000 7f48b97fa700 1 -- 172.21.15.173:0/3657035728 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f48b001c320 con 0x7f48bc0f5410 2024-02-19T22:03:06.036 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:06.036 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:06.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.038+0000 7f48c2bd9700 1 -- 172.21.15.173:0/3657035728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f48ac02ff60 msgr2=0x7f48ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:06.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.038+0000 7f48c2bd9700 1 --2- 172.21.15.173:0/3657035728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f48ac02ff60 0x7f48ac032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f48a80096a0 tx=0x7f48a8006b40).stop 2024-02-19T22:03:06.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.038+0000 7f48c2bd9700 1 -- 172.21.15.173:0/3657035728 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48bc0f5410 msgr2=0x7f48bc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:06.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.038+0000 7f48c2bd9700 1 --2- 172.21.15.173:0/3657035728 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48bc0f5410 0x7f48bc1056e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f48b00089f0 tx=0x7f48b000b040).stop 2024-02-19T22:03:06.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.038+0000 7f48c2bd9700 1 -- 172.21.15.173:0/3657035728 shutdown_connections 2024-02-19T22:03:06.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.038+0000 7f48c2bd9700 1 --2- 172.21.15.173:0/3657035728 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f48ac02ff60 0x7f48ac032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:06.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.038+0000 7f48c2bd9700 1 --2- 172.21.15.173:0/3657035728 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48bc0f5410 0x7f48bc1056e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:06.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.038+0000 7f48c2bd9700 1 -- 172.21.15.173:0/3657035728 >> 172.21.15.173:0/3657035728 conn(0x7f48bc0f0b70 msgr2=0x7f48bc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:06.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.038+0000 7f48c2bd9700 1 -- 172.21.15.173:0/3657035728 shutdown_connections 2024-02-19T22:03:06.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:06.038+0000 7f48c2bd9700 1 -- 172.21.15.173:0/3657035728 wait complete. 2024-02-19T22:03:06.039 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:04.517264+0000 mgr.smithi143.bissbc (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:06.037253+0000 mon.smithi143 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.173:0/3657035728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:07.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:07.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:06.517567+0000 mgr.smithi143.bissbc (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:09.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.800+0000 7fb721dea700 1 -- 172.21.15.173:0/933588982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71c0f5810 msgr2=0x7fb71c0f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:09.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.800+0000 7fb721dea700 1 --2- 172.21.15.173:0/933588982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71c0f5810 0x7fb71c0f5c30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fb710009fb0 tx=0x7fb710009b30).stop 2024-02-19T22:03:09.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.801+0000 7fb721dea700 1 -- 172.21.15.173:0/933588982 shutdown_connections 2024-02-19T22:03:09.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.801+0000 7fb721dea700 1 --2- 172.21.15.173:0/933588982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71c0f5810 0x7fb71c0f5c30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:09.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.801+0000 7fb721dea700 1 -- 172.21.15.173:0/933588982 >> 172.21.15.173:0/933588982 conn(0x7fb71c0f0120 msgr2=0x7fb71c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:09.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.801+0000 7fb721dea700 1 -- 172.21.15.173:0/933588982 shutdown_connections 2024-02-19T22:03:09.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.801+0000 7fb721dea700 1 -- 172.21.15.173:0/933588982 wait complete. 2024-02-19T22:03:09.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.801+0000 7fb721dea700 1 Processor -- start 2024-02-19T22:03:09.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.801+0000 7fb721dea700 1 -- start start 2024-02-19T22:03:09.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.801+0000 7fb721dea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71c0f5810 0x7fb71c109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:09.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.801+0000 7fb721dea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c0f5230 con 0x7fb71c0f5810 2024-02-19T22:03:09.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.802+0000 7fb720de8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71c0f5810 0x7fb71c109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:09.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.802+0000 7fb720de8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71c0f5810 0x7fb71c109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41504/0 (socket says 172.21.15.173:41504) 2024-02-19T22:03:09.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.802+0000 7fb720de8700 1 -- 172.21.15.173:0/4216301899 learned_addr learned my addr 172.21.15.173:0/4216301899 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:09.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.802+0000 7fb720de8700 1 -- 172.21.15.173:0/4216301899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb710005040 con 0x7fb71c0f5810 2024-02-19T22:03:09.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.802+0000 7fb720de8700 1 --2- 172.21.15.173:0/4216301899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71c0f5810 0x7fb71c109290 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fb710007a10 tx=0x7fb710009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:09.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.802+0000 7fb7197fa700 1 -- 172.21.15.173:0/4216301899 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb710018070 con 0x7fb71c0f5810 2024-02-19T22:03:09.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.803+0000 7fb721dea700 1 -- 172.21.15.173:0/4216301899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71c0f5430 con 0x7fb71c0f5810 2024-02-19T22:03:09.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.803+0000 7fb7197fa700 1 -- 172.21.15.173:0/4216301899 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb710013410 con 0x7fb71c0f5810 2024-02-19T22:03:09.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.803+0000 7fb7197fa700 1 -- 172.21.15.173:0/4216301899 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb71001c3c0 con 0x7fb71c0f5810 2024-02-19T22:03:09.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.803+0000 7fb721dea700 1 -- 172.21.15.173:0/4216301899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71c1099c0 con 0x7fb71c0f5810 2024-02-19T22:03:09.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.803+0000 7fb7197fa700 1 -- 172.21.15.173:0/4216301899 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb71001d420 con 0x7fb71c0f5810 2024-02-19T22:03:09.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.803+0000 7fb721dea700 1 -- 172.21.15.173:0/4216301899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71c045ad0 con 0x7fb71c0f5810 2024-02-19T22:03:09.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.803+0000 7fb7197fa700 1 --2- 172.21.15.173:0/4216301899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb70c02ff60 0x7fb70c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:09.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.804+0000 7fb7197fa700 1 -- 172.21.15.173:0/4216301899 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb7100415f0 con 0x7fb71c0f5810 2024-02-19T22:03:09.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.804+0000 7fb71bfff700 1 --2- 172.21.15.173:0/4216301899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb70c02ff60 0x7fb70c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:09.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.804+0000 7fb71bfff700 1 --2- 172.21.15.173:0/4216301899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb70c02ff60 0x7fb70c032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb7080096a0 tx=0x7fb708006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:09.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:09.808+0000 7fb7197fa700 1 -- 172.21.15.173:0/4216301899 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb71001d820 con 0x7fb71c0f5810 2024-02-19T22:03:10.014 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.015+0000 7fb721dea700 1 -- 172.21.15.173:0/4216301899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb71c0fad40 con 0x7fb71c0f5810 2024-02-19T22:03:10.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.015+0000 7fb7197fa700 1 -- 172.21.15.173:0/4216301899 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb71001b370 con 0x7fb71c0f5810 2024-02-19T22:03:10.015 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:10.016 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:10.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.017+0000 7fb721dea700 1 -- 172.21.15.173:0/4216301899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb70c02ff60 msgr2=0x7fb70c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:10.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.017+0000 7fb721dea700 1 --2- 172.21.15.173:0/4216301899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb70c02ff60 0x7fb70c032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb7080096a0 tx=0x7fb708006b40).stop 2024-02-19T22:03:10.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.017+0000 7fb721dea700 1 -- 172.21.15.173:0/4216301899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71c0f5810 msgr2=0x7fb71c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:10.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.017+0000 7fb721dea700 1 --2- 172.21.15.173:0/4216301899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71c0f5810 0x7fb71c109290 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fb710007a10 tx=0x7fb710009b30).stop 2024-02-19T22:03:10.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.017+0000 7fb721dea700 1 -- 172.21.15.173:0/4216301899 shutdown_connections 2024-02-19T22:03:10.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.017+0000 7fb721dea700 1 --2- 172.21.15.173:0/4216301899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb70c02ff60 0x7fb70c032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:10.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.017+0000 7fb721dea700 1 --2- 172.21.15.173:0/4216301899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb71c0f5810 0x7fb71c109290 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:10.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.017+0000 7fb721dea700 1 -- 172.21.15.173:0/4216301899 >> 172.21.15.173:0/4216301899 conn(0x7fb71c0f0120 msgr2=0x7fb71c0f0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:10.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.017+0000 7fb721dea700 1 -- 172.21.15.173:0/4216301899 shutdown_connections 2024-02-19T22:03:10.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:10.017+0000 7fb721dea700 1 -- 172.21.15.173:0/4216301899 wait complete. 2024-02-19T22:03:10.018 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:08.517884+0000 mgr.smithi143.bissbc (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:10.016459+0000 mon.smithi143 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.173:0/4216301899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:11.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:11.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:12.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:10.518175+0000 mgr.smithi143.bissbc (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:13.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.775+0000 7f7c94f76700 1 -- 172.21.15.173:0/2445976680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c900f5410 msgr2=0x7f7c900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:13.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.775+0000 7f7c94f76700 1 --2- 172.21.15.173:0/2445976680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c900f5410 0x7f7c900f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f7c80009fb0 tx=0x7f7c80009b30).stop 2024-02-19T22:03:13.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.775+0000 7f7c94f76700 1 -- 172.21.15.173:0/2445976680 shutdown_connections 2024-02-19T22:03:13.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.775+0000 7f7c94f76700 1 --2- 172.21.15.173:0/2445976680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c900f5410 0x7f7c900f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:13.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.775+0000 7f7c94f76700 1 -- 172.21.15.173:0/2445976680 >> 172.21.15.173:0/2445976680 conn(0x7f7c900f0b70 msgr2=0x7f7c900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:13.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.775+0000 7f7c94f76700 1 -- 172.21.15.173:0/2445976680 shutdown_connections 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.775+0000 7f7c94f76700 1 -- 172.21.15.173:0/2445976680 wait complete. 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.775+0000 7f7c94f76700 1 Processor -- start 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c94f76700 1 -- start start 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c94f76700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c900f5410 0x7f7c90106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c94f76700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c90106d00 con 0x7f7c900f5410 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c8e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c900f5410 0x7f7c90106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c8e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c900f5410 0x7f7c90106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:40454/0 (socket says 172.21.15.173:40454) 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c8e59c700 1 -- 172.21.15.173:0/179745310 learned_addr learned my addr 172.21.15.173:0/179745310 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c8e59c700 1 -- 172.21.15.173:0/179745310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c80005040 con 0x7f7c900f5410 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c8e59c700 1 --2- 172.21.15.173:0/179745310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c900f5410 0x7f7c90106810 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f7c80005010 tx=0x7f7c8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c7effd700 1 -- 172.21.15.173:0/179745310 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c80009320 con 0x7f7c900f5410 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c94f76700 1 -- 172.21.15.173:0/179745310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c90106f00 con 0x7f7c900f5410 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c7effd700 1 -- 172.21.15.173:0/179745310 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c80013380 con 0x7f7c900f5410 2024-02-19T22:03:13.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.776+0000 7f7c7effd700 1 -- 172.21.15.173:0/179745310 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c80014e50 con 0x7f7c900f5410 2024-02-19T22:03:13.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.777+0000 7f7c94f76700 1 -- 172.21.15.173:0/179745310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c90107390 con 0x7f7c900f5410 2024-02-19T22:03:13.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.777+0000 7f7c7effd700 1 -- 172.21.15.173:0/179745310 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c800095f0 con 0x7f7c900f5410 2024-02-19T22:03:13.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.777+0000 7f7c94f76700 1 -- 172.21.15.173:0/179745310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c90045ad0 con 0x7f7c900f5410 2024-02-19T22:03:13.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.777+0000 7f7c7effd700 1 --2- 172.21.15.173:0/179745310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7c7802ff10 0x7f7c780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:13.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.777+0000 7f7c7effd700 1 -- 172.21.15.173:0/179745310 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c80040e70 con 0x7f7c900f5410 2024-02-19T22:03:13.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.778+0000 7f7c8dd9b700 1 --2- 172.21.15.173:0/179745310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7c7802ff10 0x7f7c780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:13.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.778+0000 7f7c8dd9b700 1 --2- 172.21.15.173:0/179745310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7c7802ff10 0x7f7c780323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7c840096a0 tx=0x7f7c84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:13.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.782+0000 7f7c7effd700 1 -- 172.21.15.173:0/179745310 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c80014420 con 0x7f7c900f5410 2024-02-19T22:03:13.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.988+0000 7f7c94f76700 1 -- 172.21.15.173:0/179745310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c900fafa0 con 0x7f7c900f5410 2024-02-19T22:03:13.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.989+0000 7f7c7effd700 1 -- 172.21.15.173:0/179745310 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7c80010230 con 0x7f7c900f5410 2024-02-19T22:03:13.989 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:13.989 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:13.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.991+0000 7f7c94f76700 1 -- 172.21.15.173:0/179745310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7c7802ff10 msgr2=0x7f7c780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:13.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.991+0000 7f7c94f76700 1 --2- 172.21.15.173:0/179745310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7c7802ff10 0x7f7c780323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7c840096a0 tx=0x7f7c84006b40).stop 2024-02-19T22:03:13.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.991+0000 7f7c94f76700 1 -- 172.21.15.173:0/179745310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c900f5410 msgr2=0x7f7c90106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:13.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.991+0000 7f7c94f76700 1 --2- 172.21.15.173:0/179745310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c900f5410 0x7f7c90106810 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f7c80005010 tx=0x7f7c8000b040).stop 2024-02-19T22:03:13.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.991+0000 7f7c94f76700 1 -- 172.21.15.173:0/179745310 shutdown_connections 2024-02-19T22:03:13.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.991+0000 7f7c94f76700 1 --2- 172.21.15.173:0/179745310 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f7c7802ff10 0x7f7c780323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:13.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.991+0000 7f7c94f76700 1 --2- 172.21.15.173:0/179745310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c900f5410 0x7f7c90106810 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:13.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.991+0000 7f7c94f76700 1 -- 172.21.15.173:0/179745310 >> 172.21.15.173:0/179745310 conn(0x7f7c900f0b70 msgr2=0x7f7c9005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:13.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.991+0000 7f7c94f76700 1 -- 172.21.15.173:0/179745310 shutdown_connections 2024-02-19T22:03:13.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:13.991+0000 7f7c94f76700 1 -- 172.21.15.173:0/179745310 wait complete. 2024-02-19T22:03:13.992 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:12.518452+0000 mgr.smithi143.bissbc (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:13.990313+0000 mon.smithi143 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.173:0/179745310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:15.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:15.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:14.518780+0000 mgr.smithi143.bissbc (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:17.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.819+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4060717701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e80f5410 msgr2=0x7fe1e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:17.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.819+0000 7fe1f01a0700 1 --2- 172.21.15.173:0/4060717701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e80f5410 0x7fe1e80f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fe1d8009fb0 tx=0x7fe1d8009b30).stop 2024-02-19T22:03:17.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.819+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4060717701 shutdown_connections 2024-02-19T22:03:17.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.819+0000 7fe1f01a0700 1 --2- 172.21.15.173:0/4060717701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e80f5410 0x7fe1e80f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:17.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.819+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4060717701 >> 172.21.15.173:0/4060717701 conn(0x7fe1e80f0b70 msgr2=0x7fe1e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:17.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.819+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4060717701 shutdown_connections 2024-02-19T22:03:17.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.820+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4060717701 wait complete. 2024-02-19T22:03:17.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.820+0000 7fe1f01a0700 1 Processor -- start 2024-02-19T22:03:17.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.820+0000 7fe1f01a0700 1 -- start start 2024-02-19T22:03:17.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.820+0000 7fe1f01a0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e80f5410 0x7fe1e81094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:17.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.820+0000 7fe1f01a0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e8060280 con 0x7fe1e80f5410 2024-02-19T22:03:17.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.821+0000 7fe1edf3c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e80f5410 0x7fe1e81094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:17.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.821+0000 7fe1edf3c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e80f5410 0x7fe1e81094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:40468/0 (socket says 172.21.15.173:40468) 2024-02-19T22:03:17.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.821+0000 7fe1edf3c700 1 -- 172.21.15.173:0/4183704435 learned_addr learned my addr 172.21.15.173:0/4183704435 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.821+0000 7fe1edf3c700 1 -- 172.21.15.173:0/4183704435 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d8005040 con 0x7fe1e80f5410 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.821+0000 7fe1edf3c700 1 --2- 172.21.15.173:0/4183704435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e80f5410 0x7fe1e81094a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fe1d8005010 tx=0x7fe1d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.821+0000 7fe1de7fc700 1 -- 172.21.15.173:0/4183704435 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1d8009320 con 0x7fe1e80f5410 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.821+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4183704435 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e8060480 con 0x7fe1e80f5410 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.821+0000 7fe1de7fc700 1 -- 172.21.15.173:0/4183704435 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1d8013380 con 0x7fe1e80f5410 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.821+0000 7fe1de7fc700 1 -- 172.21.15.173:0/4183704435 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1d8014e50 con 0x7fe1e80f5410 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.821+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4183704435 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e8109bd0 con 0x7fe1e80f5410 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.822+0000 7fe1de7fc700 1 -- 172.21.15.173:0/4183704435 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1d80095f0 con 0x7fe1e80f5410 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.822+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4183704435 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e8045ad0 con 0x7fe1e80f5410 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.822+0000 7fe1de7fc700 1 --2- 172.21.15.173:0/4183704435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe1d402ff10 0x7fe1d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.822+0000 7fe1de7fc700 1 -- 172.21.15.173:0/4183704435 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1d8040f80 con 0x7fe1e80f5410 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.823+0000 7fe1ed73b700 1 --2- 172.21.15.173:0/4183704435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe1d402ff10 0x7fe1d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.823+0000 7fe1ed73b700 1 --2- 172.21.15.173:0/4183704435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe1d402ff10 0x7fe1d40323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe1e40096a0 tx=0x7fe1e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:17.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:17.827+0000 7fe1de7fc700 1 -- 172.21.15.173:0/4183704435 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1d8014420 con 0x7fe1e80f5410 2024-02-19T22:03:18.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.034+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4183704435 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1e80fafa0 con 0x7fe1e80f5410 2024-02-19T22:03:18.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.034+0000 7fe1de7fc700 1 -- 172.21.15.173:0/4183704435 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe1d8010230 con 0x7fe1e80f5410 2024-02-19T22:03:18.034 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:18.034 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:18.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.036+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4183704435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe1d402ff10 msgr2=0x7fe1d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:18.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.036+0000 7fe1f01a0700 1 --2- 172.21.15.173:0/4183704435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe1d402ff10 0x7fe1d40323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fe1e40096a0 tx=0x7fe1e4006b40).stop 2024-02-19T22:03:18.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.036+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4183704435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e80f5410 msgr2=0x7fe1e81094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:18.035 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.036+0000 7fe1f01a0700 1 --2- 172.21.15.173:0/4183704435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e80f5410 0x7fe1e81094a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fe1d8005010 tx=0x7fe1d800b040).stop 2024-02-19T22:03:18.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.036+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4183704435 shutdown_connections 2024-02-19T22:03:18.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.036+0000 7fe1f01a0700 1 --2- 172.21.15.173:0/4183704435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe1d402ff10 0x7fe1d40323d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:18.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.036+0000 7fe1f01a0700 1 --2- 172.21.15.173:0/4183704435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e80f5410 0x7fe1e81094a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:18.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.036+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4183704435 >> 172.21.15.173:0/4183704435 conn(0x7fe1e80f0b70 msgr2=0x7fe1e80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:18.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.036+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4183704435 shutdown_connections 2024-02-19T22:03:18.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:18.036+0000 7fe1f01a0700 1 -- 172.21.15.173:0/4183704435 wait complete. 2024-02-19T22:03:18.036 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:16.519030+0000 mgr.smithi143.bissbc (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:18.035326+0000 mon.smithi143 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.173:0/4183704435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:19.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:19.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:18.519335+0000 mgr.smithi143.bissbc (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:21.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.680+0000 7f015bfff700 1 -- 172.21.15.173:0/2323182594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f015c0601e0 msgr2=0x7f015c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:21.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.680+0000 7f015bfff700 1 --2- 172.21.15.173:0/2323182594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f015c0601e0 0x7f015c0f8f20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f0150009fb0 tx=0x7f0150009b30).stop 2024-02-19T22:03:21.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.680+0000 7f015bfff700 1 -- 172.21.15.173:0/2323182594 shutdown_connections 2024-02-19T22:03:21.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.680+0000 7f015bfff700 1 --2- 172.21.15.173:0/2323182594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f015c0601e0 0x7f015c0f8f20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:21.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.680+0000 7f015bfff700 1 -- 172.21.15.173:0/2323182594 >> 172.21.15.173:0/2323182594 conn(0x7f015c0f09a0 msgr2=0x7f015c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:21.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.680+0000 7f015bfff700 1 -- 172.21.15.173:0/2323182594 shutdown_connections 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.680+0000 7f015bfff700 1 -- 172.21.15.173:0/2323182594 wait complete. 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.680+0000 7f015bfff700 1 Processor -- start 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.680+0000 7f015bfff700 1 -- start start 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.681+0000 7f015bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f015c0601e0 0x7f015c106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.681+0000 7f015bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c106af0 con 0x7f015c0601e0 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.681+0000 7f015affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f015c0601e0 0x7f015c106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.681+0000 7f015affd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f015c0601e0 0x7f015c106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:35106/0 (socket says 172.21.15.173:35106) 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.681+0000 7f015affd700 1 -- 172.21.15.173:0/264533981 learned_addr learned my addr 172.21.15.173:0/264533981 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.681+0000 7f015affd700 1 -- 172.21.15.173:0/264533981 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0150005040 con 0x7f015c0601e0 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.681+0000 7f015affd700 1 --2- 172.21.15.173:0/264533981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f015c0601e0 0x7f015c106600 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f0150005010 tx=0x7f015000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.681+0000 7f01437fe700 1 -- 172.21.15.173:0/264533981 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0150009320 con 0x7f015c0601e0 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.681+0000 7f015bfff700 1 -- 172.21.15.173:0/264533981 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c106cf0 con 0x7f015c0601e0 2024-02-19T22:03:21.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.681+0000 7f01437fe700 1 -- 172.21.15.173:0/264533981 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0150013380 con 0x7f015c0601e0 2024-02-19T22:03:21.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.681+0000 7f01437fe700 1 -- 172.21.15.173:0/264533981 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0150014e50 con 0x7f015c0601e0 2024-02-19T22:03:21.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.682+0000 7f015bfff700 1 -- 172.21.15.173:0/264533981 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c107180 con 0x7f015c0601e0 2024-02-19T22:03:21.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.682+0000 7f01437fe700 1 -- 172.21.15.173:0/264533981 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01500095f0 con 0x7f015c0601e0 2024-02-19T22:03:21.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.682+0000 7f015bfff700 1 -- 172.21.15.173:0/264533981 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c045ad0 con 0x7f015c0601e0 2024-02-19T22:03:21.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.682+0000 7f01437fe700 1 --2- 172.21.15.173:0/264533981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f014402ff10 0x7f01440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:21.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.682+0000 7f01437fe700 1 -- 172.21.15.173:0/264533981 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0150040f80 con 0x7f015c0601e0 2024-02-19T22:03:21.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.683+0000 7f015a7fc700 1 --2- 172.21.15.173:0/264533981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f014402ff10 0x7f01440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:21.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.683+0000 7f015a7fc700 1 --2- 172.21.15.173:0/264533981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f014402ff10 0x7f01440323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f014c0096a0 tx=0x7f014c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:21.687 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.687+0000 7f01437fe700 1 -- 172.21.15.173:0/264533981 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0150014420 con 0x7f015c0601e0 2024-02-19T22:03:21.888 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.889+0000 7f015bfff700 1 -- 172.21.15.173:0/264533981 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f015c107fd0 con 0x7f015c0601e0 2024-02-19T22:03:21.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.890+0000 7f01437fe700 1 -- 172.21.15.173:0/264533981 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0150010230 con 0x7f015c0601e0 2024-02-19T22:03:21.889 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:21.889 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:21.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.891+0000 7f015bfff700 1 -- 172.21.15.173:0/264533981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f014402ff10 msgr2=0x7f01440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:21.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.891+0000 7f015bfff700 1 --2- 172.21.15.173:0/264533981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f014402ff10 0x7f01440323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f014c0096a0 tx=0x7f014c006b40).stop 2024-02-19T22:03:21.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.891+0000 7f015bfff700 1 -- 172.21.15.173:0/264533981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f015c0601e0 msgr2=0x7f015c106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:21.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.891+0000 7f015bfff700 1 --2- 172.21.15.173:0/264533981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f015c0601e0 0x7f015c106600 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f0150005010 tx=0x7f015000b040).stop 2024-02-19T22:03:21.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.891+0000 7f015bfff700 1 -- 172.21.15.173:0/264533981 shutdown_connections 2024-02-19T22:03:21.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.891+0000 7f015bfff700 1 --2- 172.21.15.173:0/264533981 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f014402ff10 0x7f01440323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:21.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.891+0000 7f015bfff700 1 --2- 172.21.15.173:0/264533981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f015c0601e0 0x7f015c106600 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:21.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.891+0000 7f015bfff700 1 -- 172.21.15.173:0/264533981 >> 172.21.15.173:0/264533981 conn(0x7f015c0f09a0 msgr2=0x7f015c0f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:21.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.891+0000 7f015bfff700 1 -- 172.21.15.173:0/264533981 shutdown_connections 2024-02-19T22:03:21.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:21.891+0000 7f015bfff700 1 -- 172.21.15.173:0/264533981 wait complete. 2024-02-19T22:03:21.892 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:20.571390+0000 mon.smithi143 (mon.0) 271 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T22:03:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:20.571588+0000 mon.smithi143 (mon.0) 272 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:03:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:20.519668+0000 mgr.smithi143.bissbc (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:21.890872+0000 mon.smithi143 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.173:0/264533981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:23.645 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:23.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:24.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:22.520044+0000 mgr.smithi143.bissbc (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:25.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.738+0000 7f8def59e700 1 -- 172.21.15.173:0/1856831191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df00f51a0 msgr2=0x7f8df00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:25.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.738+0000 7f8def59e700 1 --2- 172.21.15.173:0/1856831191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df00f51a0 0x7f8df00f55c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f8de0009fb0 tx=0x7f8de0009b30).stop 2024-02-19T22:03:25.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.738+0000 7f8def59e700 1 -- 172.21.15.173:0/1856831191 shutdown_connections 2024-02-19T22:03:25.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.738+0000 7f8def59e700 1 --2- 172.21.15.173:0/1856831191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df00f51a0 0x7f8df00f55c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:25.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.738+0000 7f8def59e700 1 -- 172.21.15.173:0/1856831191 >> 172.21.15.173:0/1856831191 conn(0x7f8df00f09a0 msgr2=0x7f8df00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:25.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.738+0000 7f8def59e700 1 -- 172.21.15.173:0/1856831191 shutdown_connections 2024-02-19T22:03:25.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.738+0000 7f8def59e700 1 -- 172.21.15.173:0/1856831191 wait complete. 2024-02-19T22:03:25.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.739+0000 7f8def59e700 1 Processor -- start 2024-02-19T22:03:25.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.739+0000 7f8def59e700 1 -- start start 2024-02-19T22:03:25.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.739+0000 7f8def59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df00f51a0 0x7f8df01054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:25.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.739+0000 7f8def59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df01059c0 con 0x7f8df00f51a0 2024-02-19T22:03:25.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.739+0000 7f8dee59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df00f51a0 0x7f8df01054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:25.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.739+0000 7f8dee59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df00f51a0 0x7f8df01054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:35118/0 (socket says 172.21.15.173:35118) 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.739+0000 7f8dee59c700 1 -- 172.21.15.173:0/55155411 learned_addr learned my addr 172.21.15.173:0/55155411 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.739+0000 7f8dee59c700 1 -- 172.21.15.173:0/55155411 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de0005040 con 0x7f8df00f51a0 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.739+0000 7f8dee59c700 1 --2- 172.21.15.173:0/55155411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df00f51a0 0x7f8df01054d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f8de00089f0 tx=0x7f8de000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.740+0000 7f8ddeffd700 1 -- 172.21.15.173:0/55155411 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8de0009320 con 0x7f8df00f51a0 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.740+0000 7f8ddeffd700 1 -- 172.21.15.173:0/55155411 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8de0013380 con 0x7f8df00f51a0 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.740+0000 7f8def59e700 1 -- 172.21.15.173:0/55155411 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df0105bc0 con 0x7f8df00f51a0 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.740+0000 7f8ddeffd700 1 -- 172.21.15.173:0/55155411 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8de0014e20 con 0x7f8df00f51a0 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.740+0000 7f8def59e700 1 -- 172.21.15.173:0/55155411 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df0106050 con 0x7f8df00f51a0 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.740+0000 7f8ddeffd700 1 -- 172.21.15.173:0/55155411 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8de0014420 con 0x7f8df00f51a0 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.740+0000 7f8def59e700 1 -- 172.21.15.173:0/55155411 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df0045ad0 con 0x7f8df00f51a0 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.740+0000 7f8ddeffd700 1 --2- 172.21.15.173:0/55155411 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8dd802ff10 0x7f8dd80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.741+0000 7f8ddeffd700 1 -- 172.21.15.173:0/55155411 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8de0040fc0 con 0x7f8df00f51a0 2024-02-19T22:03:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.741+0000 7f8dedd9b700 1 --2- 172.21.15.173:0/55155411 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8dd802ff10 0x7f8dd80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:25.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.741+0000 7f8dedd9b700 1 --2- 172.21.15.173:0/55155411 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8dd802ff10 0x7f8dd80323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f8de40096a0 tx=0x7f8de4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:25.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.746+0000 7f8ddeffd700 1 -- 172.21.15.173:0/55155411 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8de00095f0 con 0x7f8df00f51a0 2024-02-19T22:03:25.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.952+0000 7f8def59e700 1 -- 172.21.15.173:0/55155411 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8df0106f00 con 0x7f8df00f51a0 2024-02-19T22:03:25.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.953+0000 7f8ddeffd700 1 -- 172.21.15.173:0/55155411 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8de001db70 con 0x7f8df00f51a0 2024-02-19T22:03:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:25.953 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:25.954 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.955+0000 7f8def59e700 1 -- 172.21.15.173:0/55155411 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8dd802ff10 msgr2=0x7f8dd80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:25.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.955+0000 7f8def59e700 1 --2- 172.21.15.173:0/55155411 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8dd802ff10 0x7f8dd80323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f8de40096a0 tx=0x7f8de4006b40).stop 2024-02-19T22:03:25.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.955+0000 7f8def59e700 1 -- 172.21.15.173:0/55155411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df00f51a0 msgr2=0x7f8df01054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:25.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.955+0000 7f8def59e700 1 --2- 172.21.15.173:0/55155411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df00f51a0 0x7f8df01054d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f8de00089f0 tx=0x7f8de000b040).stop 2024-02-19T22:03:25.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.955+0000 7f8def59e700 1 -- 172.21.15.173:0/55155411 shutdown_connections 2024-02-19T22:03:25.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.955+0000 7f8def59e700 1 --2- 172.21.15.173:0/55155411 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8dd802ff10 0x7f8dd80323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:25.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.955+0000 7f8def59e700 1 --2- 172.21.15.173:0/55155411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8df00f51a0 0x7f8df01054d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:25.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.955+0000 7f8def59e700 1 -- 172.21.15.173:0/55155411 >> 172.21.15.173:0/55155411 conn(0x7f8df00f09a0 msgr2=0x7f8df00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:25.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.955+0000 7f8def59e700 1 -- 172.21.15.173:0/55155411 shutdown_connections 2024-02-19T22:03:25.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:25.955+0000 7f8def59e700 1 -- 172.21.15.173:0/55155411 wait complete. 2024-02-19T22:03:25.956 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:24.520320+0000 mgr.smithi143.bissbc (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:25.954131+0000 mon.smithi143 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.173:0/55155411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:27.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:27.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:26.520676+0000 mgr.smithi143.bissbc (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:29.787 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.786+0000 7fda35618700 1 -- 172.21.15.173:0/1898175627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300f5410 msgr2=0x7fda300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:29.787 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.786+0000 7fda35618700 1 --2- 172.21.15.173:0/1898175627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300f5410 0x7fda300f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fda18009fb0 tx=0x7fda18009b30).stop 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.786+0000 7fda35618700 1 -- 172.21.15.173:0/1898175627 shutdown_connections 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.786+0000 7fda35618700 1 --2- 172.21.15.173:0/1898175627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300f5410 0x7fda300f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.786+0000 7fda35618700 1 -- 172.21.15.173:0/1898175627 >> 172.21.15.173:0/1898175627 conn(0x7fda300f0b70 msgr2=0x7fda300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.786+0000 7fda35618700 1 -- 172.21.15.173:0/1898175627 shutdown_connections 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.786+0000 7fda35618700 1 -- 172.21.15.173:0/1898175627 wait complete. 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.787+0000 7fda35618700 1 Processor -- start 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.787+0000 7fda35618700 1 -- start start 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.787+0000 7fda35618700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300f5410 0x7fda30109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.787+0000 7fda35618700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda30060220 con 0x7fda300f5410 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.787+0000 7fda2effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300f5410 0x7fda30109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.787+0000 7fda2effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300f5410 0x7fda30109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:35126/0 (socket says 172.21.15.173:35126) 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.787+0000 7fda2effd700 1 -- 172.21.15.173:0/3419287451 learned_addr learned my addr 172.21.15.173:0/3419287451 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:29.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.787+0000 7fda2effd700 1 -- 172.21.15.173:0/3419287451 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18005040 con 0x7fda300f5410 2024-02-19T22:03:29.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.788+0000 7fda2effd700 1 --2- 172.21.15.173:0/3419287451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300f5410 0x7fda30109540 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fda18005010 tx=0x7fda1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:29.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.788+0000 7fda277fe700 1 -- 172.21.15.173:0/3419287451 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda18009320 con 0x7fda300f5410 2024-02-19T22:03:29.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.788+0000 7fda277fe700 1 -- 172.21.15.173:0/3419287451 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda18013380 con 0x7fda300f5410 2024-02-19T22:03:29.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.788+0000 7fda35618700 1 -- 172.21.15.173:0/3419287451 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda30060420 con 0x7fda300f5410 2024-02-19T22:03:29.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.788+0000 7fda277fe700 1 -- 172.21.15.173:0/3419287451 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda18014e50 con 0x7fda300f5410 2024-02-19T22:03:29.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.788+0000 7fda35618700 1 -- 172.21.15.173:0/3419287451 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda30109bf0 con 0x7fda300f5410 2024-02-19T22:03:29.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.788+0000 7fda277fe700 1 -- 172.21.15.173:0/3419287451 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda180095f0 con 0x7fda300f5410 2024-02-19T22:03:29.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.789+0000 7fda35618700 1 -- 172.21.15.173:0/3419287451 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda30059100 con 0x7fda300f5410 2024-02-19T22:03:29.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.789+0000 7fda277fe700 1 --2- 172.21.15.173:0/3419287451 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fda1c02ff10 0x7fda1c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:29.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.789+0000 7fda277fe700 1 -- 172.21.15.173:0/3419287451 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda18040f60 con 0x7fda300f5410 2024-02-19T22:03:29.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.789+0000 7fda2e7fc700 1 --2- 172.21.15.173:0/3419287451 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fda1c02ff10 0x7fda1c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:29.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.789+0000 7fda2e7fc700 1 --2- 172.21.15.173:0/3419287451 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fda1c02ff10 0x7fda1c0323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fda200096a0 tx=0x7fda20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:29.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:29.794+0000 7fda277fe700 1 -- 172.21.15.173:0/3419287451 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda18014420 con 0x7fda300f5410 2024-02-19T22:03:30.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.000+0000 7fda35618700 1 -- 172.21.15.173:0/3419287451 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda30045ad0 con 0x7fda300f5410 2024-02-19T22:03:30.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.001+0000 7fda277fe700 1 -- 172.21.15.173:0/3419287451 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fda18010230 con 0x7fda300f5410 2024-02-19T22:03:30.001 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:30.001 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:30.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.003+0000 7fda35618700 1 -- 172.21.15.173:0/3419287451 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fda1c02ff10 msgr2=0x7fda1c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:30.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.003+0000 7fda35618700 1 --2- 172.21.15.173:0/3419287451 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fda1c02ff10 0x7fda1c0323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fda200096a0 tx=0x7fda20006b40).stop 2024-02-19T22:03:30.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.003+0000 7fda35618700 1 -- 172.21.15.173:0/3419287451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300f5410 msgr2=0x7fda30109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:30.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.003+0000 7fda35618700 1 --2- 172.21.15.173:0/3419287451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300f5410 0x7fda30109540 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fda18005010 tx=0x7fda1800b040).stop 2024-02-19T22:03:30.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.003+0000 7fda35618700 1 -- 172.21.15.173:0/3419287451 shutdown_connections 2024-02-19T22:03:30.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.003+0000 7fda35618700 1 --2- 172.21.15.173:0/3419287451 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fda1c02ff10 0x7fda1c0323d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:30.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.003+0000 7fda35618700 1 --2- 172.21.15.173:0/3419287451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300f5410 0x7fda30109540 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:30.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.003+0000 7fda35618700 1 -- 172.21.15.173:0/3419287451 >> 172.21.15.173:0/3419287451 conn(0x7fda300f0b70 msgr2=0x7fda300f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:30.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.003+0000 7fda35618700 1 -- 172.21.15.173:0/3419287451 shutdown_connections 2024-02-19T22:03:30.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:30.003+0000 7fda35618700 1 -- 172.21.15.173:0/3419287451 wait complete. 2024-02-19T22:03:30.003 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:28.520945+0000 mgr.smithi143.bissbc (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:30.002433+0000 mon.smithi143 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.173:0/3419287451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:31.640 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:31.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:32.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:30.521230+0000 mgr.smithi143.bissbc (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:33.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.733+0000 7fb4529c8700 1 -- 172.21.15.173:0/1467244076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb44c0f5410 msgr2=0x7fb44c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:33.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.733+0000 7fb4529c8700 1 --2- 172.21.15.173:0/1467244076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb44c0f5410 0x7fb44c0f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fb43c009fb0 tx=0x7fb43c009b30).stop 2024-02-19T22:03:33.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.733+0000 7fb4529c8700 1 -- 172.21.15.173:0/1467244076 shutdown_connections 2024-02-19T22:03:33.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.733+0000 7fb4529c8700 1 --2- 172.21.15.173:0/1467244076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb44c0f5410 0x7fb44c0f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:33.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.733+0000 7fb4529c8700 1 -- 172.21.15.173:0/1467244076 >> 172.21.15.173:0/1467244076 conn(0x7fb44c0f0b70 msgr2=0x7fb44c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:33.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.733+0000 7fb4529c8700 1 -- 172.21.15.173:0/1467244076 shutdown_connections 2024-02-19T22:03:33.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.733+0000 7fb4529c8700 1 -- 172.21.15.173:0/1467244076 wait complete. 2024-02-19T22:03:33.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.734+0000 7fb4529c8700 1 Processor -- start 2024-02-19T22:03:33.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.734+0000 7fb4529c8700 1 -- start start 2024-02-19T22:03:33.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.734+0000 7fb4529c8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb44c0f5410 0x7fb44c109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:33.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.734+0000 7fb4529c8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c060220 con 0x7fb44c0f5410 2024-02-19T22:03:33.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.734+0000 7fb44bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb44c0f5410 0x7fb44c109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:33.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.734+0000 7fb44bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb44c0f5410 0x7fb44c109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:56012/0 (socket says 172.21.15.173:56012) 2024-02-19T22:03:33.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.734+0000 7fb44bfff700 1 -- 172.21.15.173:0/2113737931 learned_addr learned my addr 172.21.15.173:0/2113737931 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:33.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.735+0000 7fb44bfff700 1 -- 172.21.15.173:0/2113737931 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43c005040 con 0x7fb44c0f5410 2024-02-19T22:03:33.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.735+0000 7fb44bfff700 1 --2- 172.21.15.173:0/2113737931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb44c0f5410 0x7fb44c109540 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fb43c005010 tx=0x7fb43c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:33.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.735+0000 7fb448ff9700 1 -- 172.21.15.173:0/2113737931 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb43c009320 con 0x7fb44c0f5410 2024-02-19T22:03:33.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.735+0000 7fb4529c8700 1 -- 172.21.15.173:0/2113737931 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c060420 con 0x7fb44c0f5410 2024-02-19T22:03:33.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.735+0000 7fb448ff9700 1 -- 172.21.15.173:0/2113737931 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb43c013380 con 0x7fb44c0f5410 2024-02-19T22:03:33.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.735+0000 7fb448ff9700 1 -- 172.21.15.173:0/2113737931 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb43c014e50 con 0x7fb44c0f5410 2024-02-19T22:03:33.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.735+0000 7fb4529c8700 1 -- 172.21.15.173:0/2113737931 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44c109bf0 con 0x7fb44c0f5410 2024-02-19T22:03:33.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.736+0000 7fb448ff9700 1 -- 172.21.15.173:0/2113737931 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb43c014420 con 0x7fb44c0f5410 2024-02-19T22:03:33.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.736+0000 7fb4529c8700 1 -- 172.21.15.173:0/2113737931 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44c059100 con 0x7fb44c0f5410 2024-02-19T22:03:33.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.736+0000 7fb448ff9700 1 --2- 172.21.15.173:0/2113737931 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb43402ff10 0x7fb4340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:33.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.736+0000 7fb448ff9700 1 -- 172.21.15.173:0/2113737931 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb43c040f70 con 0x7fb44c0f5410 2024-02-19T22:03:33.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.736+0000 7fb44b7fe700 1 --2- 172.21.15.173:0/2113737931 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb43402ff10 0x7fb4340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:33.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.737+0000 7fb44b7fe700 1 --2- 172.21.15.173:0/2113737931 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb43402ff10 0x7fb4340323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb4400096a0 tx=0x7fb440006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:33.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.741+0000 7fb448ff9700 1 -- 172.21.15.173:0/2113737931 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb43c0095f0 con 0x7fb44c0f5410 2024-02-19T22:03:33.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.947+0000 7fb4529c8700 1 -- 172.21.15.173:0/2113737931 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb44c045ad0 con 0x7fb44c0f5410 2024-02-19T22:03:33.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.948+0000 7fb448ff9700 1 -- 172.21.15.173:0/2113737931 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb43c01db70 con 0x7fb44c0f5410 2024-02-19T22:03:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:33.948 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:33.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.949+0000 7fb4529c8700 1 -- 172.21.15.173:0/2113737931 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb43402ff10 msgr2=0x7fb4340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:33.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.949+0000 7fb4529c8700 1 --2- 172.21.15.173:0/2113737931 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb43402ff10 0x7fb4340323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb4400096a0 tx=0x7fb440006b40).stop 2024-02-19T22:03:33.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.949+0000 7fb4529c8700 1 -- 172.21.15.173:0/2113737931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb44c0f5410 msgr2=0x7fb44c109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:33.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.949+0000 7fb4529c8700 1 --2- 172.21.15.173:0/2113737931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb44c0f5410 0x7fb44c109540 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fb43c005010 tx=0x7fb43c00b040).stop 2024-02-19T22:03:33.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.949+0000 7fb4529c8700 1 -- 172.21.15.173:0/2113737931 shutdown_connections 2024-02-19T22:03:33.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.949+0000 7fb4529c8700 1 --2- 172.21.15.173:0/2113737931 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb43402ff10 0x7fb4340323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:33.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.949+0000 7fb4529c8700 1 --2- 172.21.15.173:0/2113737931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb44c0f5410 0x7fb44c109540 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:33.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.949+0000 7fb4529c8700 1 -- 172.21.15.173:0/2113737931 >> 172.21.15.173:0/2113737931 conn(0x7fb44c0f0b70 msgr2=0x7fb44c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:33.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.949+0000 7fb4529c8700 1 -- 172.21.15.173:0/2113737931 shutdown_connections 2024-02-19T22:03:33.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:33.949+0000 7fb4529c8700 1 -- 172.21.15.173:0/2113737931 wait complete. 2024-02-19T22:03:33.950 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:32.521493+0000 mgr.smithi143.bissbc (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:33.949076+0000 mon.smithi143 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.173:0/2113737931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:35.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:35.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:34.521761+0000 mgr.smithi143.bissbc (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:37.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.653+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3422038596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0b40f5410 msgr2=0x7fd0b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:37.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.653+0000 7fd0bbe90700 1 --2- 172.21.15.173:0/3422038596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0b40f5410 0x7fd0b40f5830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fd0a8009fb0 tx=0x7fd0a8009b30).stop 2024-02-19T22:03:37.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.653+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3422038596 shutdown_connections 2024-02-19T22:03:37.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.653+0000 7fd0bbe90700 1 --2- 172.21.15.173:0/3422038596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0b40f5410 0x7fd0b40f5830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.653+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3422038596 >> 172.21.15.173:0/3422038596 conn(0x7fd0b40f0b70 msgr2=0x7fd0b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.653+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3422038596 shutdown_connections 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.653+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3422038596 wait complete. 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.654+0000 7fd0bbe90700 1 Processor -- start 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.654+0000 7fd0bbe90700 1 -- start start 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.654+0000 7fd0bbe90700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0b40f5410 0x7fd0b41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.654+0000 7fd0bbe90700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b4105bd0 con 0x7fd0b40f5410 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.654+0000 7fd0b9c2c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0b40f5410 0x7fd0b41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.654+0000 7fd0b9c2c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0b40f5410 0x7fd0b41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:56038/0 (socket says 172.21.15.173:56038) 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.654+0000 7fd0b9c2c700 1 -- 172.21.15.173:0/3495251209 learned_addr learned my addr 172.21.15.173:0/3495251209 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.654+0000 7fd0b9c2c700 1 -- 172.21.15.173:0/3495251209 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a8005040 con 0x7fd0b40f5410 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.654+0000 7fd0b9c2c700 1 --2- 172.21.15.173:0/3495251209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0b40f5410 0x7fd0b41056e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fd0a8005010 tx=0x7fd0a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:37.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.655+0000 7fd0a67fc700 1 -- 172.21.15.173:0/3495251209 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0a8009320 con 0x7fd0b40f5410 2024-02-19T22:03:37.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.655+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3495251209 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b4105dd0 con 0x7fd0b40f5410 2024-02-19T22:03:37.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.655+0000 7fd0a67fc700 1 -- 172.21.15.173:0/3495251209 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0a8013380 con 0x7fd0b40f5410 2024-02-19T22:03:37.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.655+0000 7fd0a67fc700 1 -- 172.21.15.173:0/3495251209 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0a8014e50 con 0x7fd0b40f5410 2024-02-19T22:03:37.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.655+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3495251209 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b4106260 con 0x7fd0b40f5410 2024-02-19T22:03:37.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.655+0000 7fd0a67fc700 1 -- 172.21.15.173:0/3495251209 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0a80263a0 con 0x7fd0b40f5410 2024-02-19T22:03:37.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.656+0000 7fd0a67fc700 1 --2- 172.21.15.173:0/3495251209 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd0a002ff10 0x7fd0a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:37.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.656+0000 7fd0a67fc700 1 -- 172.21.15.173:0/3495251209 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0a8040fa0 con 0x7fd0b40f5410 2024-02-19T22:03:37.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.656+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3495251209 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b4045ad0 con 0x7fd0b40f5410 2024-02-19T22:03:37.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.656+0000 7fd0b942b700 1 --2- 172.21.15.173:0/3495251209 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd0a002ff10 0x7fd0a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:37.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.656+0000 7fd0b942b700 1 --2- 172.21.15.173:0/3495251209 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd0a002ff10 0x7fd0a00323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fd0b00096a0 tx=0x7fd0b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:37.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.661+0000 7fd0a67fc700 1 -- 172.21.15.173:0/3495251209 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0a8012920 con 0x7fd0b40f5410 2024-02-19T22:03:37.867 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.868+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3495251209 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0b4107090 con 0x7fd0b40f5410 2024-02-19T22:03:37.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.868+0000 7fd0a67fc700 1 -- 172.21.15.173:0/3495251209 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd0a800f340 con 0x7fd0b40f5410 2024-02-19T22:03:37.868 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:37.868 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:37.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.870+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3495251209 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd0a002ff10 msgr2=0x7fd0a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:37.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.870+0000 7fd0bbe90700 1 --2- 172.21.15.173:0/3495251209 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd0a002ff10 0x7fd0a00323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fd0b00096a0 tx=0x7fd0b0006b40).stop 2024-02-19T22:03:37.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.870+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3495251209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0b40f5410 msgr2=0x7fd0b41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:37.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.870+0000 7fd0bbe90700 1 --2- 172.21.15.173:0/3495251209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0b40f5410 0x7fd0b41056e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fd0a8005010 tx=0x7fd0a800b040).stop 2024-02-19T22:03:37.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.870+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3495251209 shutdown_connections 2024-02-19T22:03:37.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.870+0000 7fd0bbe90700 1 --2- 172.21.15.173:0/3495251209 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd0a002ff10 0x7fd0a00323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:37.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.870+0000 7fd0bbe90700 1 --2- 172.21.15.173:0/3495251209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0b40f5410 0x7fd0b41056e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:37.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.870+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3495251209 >> 172.21.15.173:0/3495251209 conn(0x7fd0b40f0b70 msgr2=0x7fd0b40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:37.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.870+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3495251209 shutdown_connections 2024-02-19T22:03:37.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:37.870+0000 7fd0bbe90700 1 -- 172.21.15.173:0/3495251209 wait complete. 2024-02-19T22:03:37.871 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:36.522094+0000 mgr.smithi143.bissbc (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:37.869600+0000 mon.smithi143 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.173:0/3495251209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:39.562 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:39.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:38.522361+0000 mgr.smithi143.bissbc (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:41.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.532+0000 7f122d657700 1 -- 172.21.15.173:0/2984548600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12280f5410 msgr2=0x7f12280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:41.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.532+0000 7f122d657700 1 --2- 172.21.15.173:0/2984548600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12280f5410 0x7f12280f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f121c009fb0 tx=0x7f121c009b30).stop 2024-02-19T22:03:41.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.532+0000 7f122d657700 1 -- 172.21.15.173:0/2984548600 shutdown_connections 2024-02-19T22:03:41.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.532+0000 7f122d657700 1 --2- 172.21.15.173:0/2984548600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12280f5410 0x7f12280f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:41.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.532+0000 7f122d657700 1 -- 172.21.15.173:0/2984548600 >> 172.21.15.173:0/2984548600 conn(0x7f12280f0b70 msgr2=0x7f12280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:41.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.532+0000 7f122d657700 1 -- 172.21.15.173:0/2984548600 shutdown_connections 2024-02-19T22:03:41.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.532+0000 7f122d657700 1 -- 172.21.15.173:0/2984548600 wait complete. 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.533+0000 7f122d657700 1 Processor -- start 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.533+0000 7f122d657700 1 -- start start 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.533+0000 7f122d657700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12280f5410 0x7f12281067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.533+0000 7f122d657700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1228106cd0 con 0x7f12280f5410 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.534+0000 7f1226ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12280f5410 0x7f12281067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.534+0000 7f1226ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12280f5410 0x7f12281067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:58214/0 (socket says 172.21.15.173:58214) 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.534+0000 7f1226ffd700 1 -- 172.21.15.173:0/4291799129 learned_addr learned my addr 172.21.15.173:0/4291799129 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.534+0000 7f1226ffd700 1 -- 172.21.15.173:0/4291799129 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c005040 con 0x7f12280f5410 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.534+0000 7f1226ffd700 1 --2- 172.21.15.173:0/4291799129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12280f5410 0x7f12281067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f121c005010 tx=0x7f121c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.534+0000 7f120f7fe700 1 -- 172.21.15.173:0/4291799129 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f121c009320 con 0x7f12280f5410 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.534+0000 7f120f7fe700 1 -- 172.21.15.173:0/4291799129 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f121c013380 con 0x7f12280f5410 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.534+0000 7f122d657700 1 -- 172.21.15.173:0/4291799129 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1228106ed0 con 0x7f12280f5410 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.534+0000 7f120f7fe700 1 -- 172.21.15.173:0/4291799129 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f121c014e50 con 0x7f12280f5410 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.534+0000 7f122d657700 1 -- 172.21.15.173:0/4291799129 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1228107360 con 0x7f12280f5410 2024-02-19T22:03:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.535+0000 7f120f7fe700 1 -- 172.21.15.173:0/4291799129 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f121c014420 con 0x7f12280f5410 2024-02-19T22:03:41.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.535+0000 7f122d657700 1 -- 172.21.15.173:0/4291799129 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1228045ad0 con 0x7f12280f5410 2024-02-19T22:03:41.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.535+0000 7f120f7fe700 1 --2- 172.21.15.173:0/4291799129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f121002ff10 0x7f12100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:41.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.535+0000 7f120f7fe700 1 -- 172.21.15.173:0/4291799129 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f121c040f40 con 0x7f12280f5410 2024-02-19T22:03:41.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.535+0000 7f12267fc700 1 --2- 172.21.15.173:0/4291799129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f121002ff10 0x7f12100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:41.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.536+0000 7f12267fc700 1 --2- 172.21.15.173:0/4291799129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f121002ff10 0x7f12100323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f12180096a0 tx=0x7f1218006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:41.542 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.540+0000 7f120f7fe700 1 -- 172.21.15.173:0/4291799129 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f121c0095f0 con 0x7f12280f5410 2024-02-19T22:03:41.743 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.744+0000 7f122d657700 1 -- 172.21.15.173:0/4291799129 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1228108210 con 0x7f12280f5410 2024-02-19T22:03:41.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.744+0000 7f120f7fe700 1 -- 172.21.15.173:0/4291799129 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f121c01db70 con 0x7f12280f5410 2024-02-19T22:03:41.744 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:41.744 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:41.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.746+0000 7f122d657700 1 -- 172.21.15.173:0/4291799129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f121002ff10 msgr2=0x7f12100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:41.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.746+0000 7f122d657700 1 --2- 172.21.15.173:0/4291799129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f121002ff10 0x7f12100323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f12180096a0 tx=0x7f1218006b40).stop 2024-02-19T22:03:41.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.746+0000 7f122d657700 1 -- 172.21.15.173:0/4291799129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12280f5410 msgr2=0x7f12281067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:41.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.746+0000 7f122d657700 1 --2- 172.21.15.173:0/4291799129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12280f5410 0x7f12281067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f121c005010 tx=0x7f121c00b040).stop 2024-02-19T22:03:41.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.746+0000 7f122d657700 1 -- 172.21.15.173:0/4291799129 shutdown_connections 2024-02-19T22:03:41.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.746+0000 7f122d657700 1 --2- 172.21.15.173:0/4291799129 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f121002ff10 0x7f12100323d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:41.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.746+0000 7f122d657700 1 --2- 172.21.15.173:0/4291799129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12280f5410 0x7f12281067e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:41.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.746+0000 7f122d657700 1 -- 172.21.15.173:0/4291799129 >> 172.21.15.173:0/4291799129 conn(0x7f12280f0b70 msgr2=0x7f12280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:41.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.746+0000 7f122d657700 1 -- 172.21.15.173:0/4291799129 shutdown_connections 2024-02-19T22:03:41.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:41.746+0000 7f122d657700 1 -- 172.21.15.173:0/4291799129 wait complete. 2024-02-19T22:03:41.747 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:40.522588+0000 mgr.smithi143.bissbc (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:41.745568+0000 mon.smithi143 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.173:0/4291799129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:43.569 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:43.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:42.522877+0000 mgr.smithi143.bissbc (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.732+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3061463141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c0f5410 msgr2=0x7f3f0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.732+0000 7f3f13ca2700 1 --2- 172.21.15.173:0/3061463141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f3f08009fb0 tx=0x7f3f08009b30).stop 2024-02-19T22:03:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.732+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3061463141 shutdown_connections 2024-02-19T22:03:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.732+0000 7f3f13ca2700 1 --2- 172.21.15.173:0/3061463141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.732+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3061463141 >> 172.21.15.173:0/3061463141 conn(0x7f3f0c0f0b70 msgr2=0x7f3f0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.733+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3061463141 shutdown_connections 2024-02-19T22:03:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.733+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3061463141 wait complete. 2024-02-19T22:03:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.733+0000 7f3f13ca2700 1 Processor -- start 2024-02-19T22:03:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.733+0000 7f3f13ca2700 1 -- start start 2024-02-19T22:03:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.733+0000 7f3f13ca2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.733+0000 7f3f13ca2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0fb960 con 0x7f3f0c0f5410 2024-02-19T22:03:45.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.733+0000 7f3f11a3e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:45.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.733+0000 7f3f11a3e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:58222/0 (socket says 172.21.15.173:58222) 2024-02-19T22:03:45.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.733+0000 7f3f11a3e700 1 -- 172.21.15.173:0/3719151550 learned_addr learned my addr 172.21.15.173:0/3719151550 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:45.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.734+0000 7f3f11a3e700 1 -- 172.21.15.173:0/3719151550 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f08005040 con 0x7f3f0c0f5410 2024-02-19T22:03:45.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.734+0000 7f3f11a3e700 1 --2- 172.21.15.173:0/3719151550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fd4e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3f08005010 tx=0x7f3f0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:45.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.734+0000 7f3f027fc700 1 -- 172.21.15.173:0/3719151550 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f08009320 con 0x7f3f0c0f5410 2024-02-19T22:03:45.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.734+0000 7f3f027fc700 1 -- 172.21.15.173:0/3719151550 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f08013380 con 0x7f3f0c0f5410 2024-02-19T22:03:45.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.734+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3719151550 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c0fbb00 con 0x7f3f0c0f5410 2024-02-19T22:03:45.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.734+0000 7f3f027fc700 1 -- 172.21.15.173:0/3719151550 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f08014e50 con 0x7f3f0c0f5410 2024-02-19T22:03:45.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.734+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3719151550 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c0fbf90 con 0x7f3f0c0f5410 2024-02-19T22:03:45.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.735+0000 7f3f027fc700 1 -- 172.21.15.173:0/3719151550 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f080095f0 con 0x7f3f0c0f5410 2024-02-19T22:03:45.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.735+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3719151550 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c059100 con 0x7f3f0c0f5410 2024-02-19T22:03:45.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.735+0000 7f3f027fc700 1 --2- 172.21.15.173:0/3719151550 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ef802ff10 0x7f3ef80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:45.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.735+0000 7f3f027fc700 1 -- 172.21.15.173:0/3719151550 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f08040e70 con 0x7f3f0c0f5410 2024-02-19T22:03:45.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.735+0000 7f3f1123d700 1 --2- 172.21.15.173:0/3719151550 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ef802ff10 0x7f3ef80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:45.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.736+0000 7f3f1123d700 1 --2- 172.21.15.173:0/3719151550 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ef802ff10 0x7f3ef80323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f3efc0096a0 tx=0x7f3efc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:45.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.741+0000 7f3f027fc700 1 -- 172.21.15.173:0/3719151550 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f0801dbc0 con 0x7f3f0c0f5410 2024-02-19T22:03:45.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:45 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:44.523116+0000 mgr.smithi143.bissbc (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:45.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.946+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3719151550 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f0c0fafa0 con 0x7f3f0c0f5410 2024-02-19T22:03:45.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.947+0000 7f3f027fc700 1 -- 172.21.15.173:0/3719151550 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3f08010230 con 0x7f3f0c0f5410 2024-02-19T22:03:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:45.948 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:45.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.949+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3719151550 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ef802ff10 msgr2=0x7f3ef80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:45.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.949+0000 7f3f13ca2700 1 --2- 172.21.15.173:0/3719151550 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ef802ff10 0x7f3ef80323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f3efc0096a0 tx=0x7f3efc006b40).stop 2024-02-19T22:03:45.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.949+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3719151550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c0f5410 msgr2=0x7f3f0c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.949+0000 7f3f13ca2700 1 --2- 172.21.15.173:0/3719151550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fd4e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3f08005010 tx=0x7f3f0800b040).stop 2024-02-19T22:03:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.949+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3719151550 shutdown_connections 2024-02-19T22:03:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.949+0000 7f3f13ca2700 1 --2- 172.21.15.173:0/3719151550 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ef802ff10 0x7f3ef80323d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.949+0000 7f3f13ca2700 1 --2- 172.21.15.173:0/3719151550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f0c0f5410 0x7f3f0c0fd4e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.949+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3719151550 >> 172.21.15.173:0/3719151550 conn(0x7f3f0c0f0b70 msgr2=0x7f3f0c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.949+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3719151550 shutdown_connections 2024-02-19T22:03:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:45.949+0000 7f3f13ca2700 1 -- 172.21.15.173:0/3719151550 wait complete. 2024-02-19T22:03:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:45.949355+0000 mon.smithi143 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.173:0/3719151550' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:47.771 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:47.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:46.523377+0000 mgr.smithi143.bissbc (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:49.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.768+0000 7ff80f341700 1 -- 172.21.15.173:0/1147806543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8080f5410 msgr2=0x7ff8080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.768+0000 7ff80f341700 1 --2- 172.21.15.173:0/1147806543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8080f5410 0x7ff8080f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7ff7f8009fb0 tx=0x7ff7f8009b30).stop 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.768+0000 7ff80f341700 1 -- 172.21.15.173:0/1147806543 shutdown_connections 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.768+0000 7ff80f341700 1 --2- 172.21.15.173:0/1147806543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8080f5410 0x7ff8080f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.768+0000 7ff80f341700 1 -- 172.21.15.173:0/1147806543 >> 172.21.15.173:0/1147806543 conn(0x7ff8080f0b70 msgr2=0x7ff8080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.768+0000 7ff80f341700 1 -- 172.21.15.173:0/1147806543 shutdown_connections 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.768+0000 7ff80f341700 1 -- 172.21.15.173:0/1147806543 wait complete. 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.769+0000 7ff80f341700 1 Processor -- start 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.769+0000 7ff80f341700 1 -- start start 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.769+0000 7ff80f341700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8080f5410 0x7ff808105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.769+0000 7ff80f341700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff808105c00 con 0x7ff8080f5410 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.769+0000 7ff80d0dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8080f5410 0x7ff808105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:49.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.769+0000 7ff80d0dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8080f5410 0x7ff808105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:58240/0 (socket says 172.21.15.173:58240) 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.769+0000 7ff80d0dd700 1 -- 172.21.15.173:0/349197497 learned_addr learned my addr 172.21.15.173:0/349197497 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.769+0000 7ff80d0dd700 1 -- 172.21.15.173:0/349197497 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f8005040 con 0x7ff8080f5410 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.769+0000 7ff80d0dd700 1 --2- 172.21.15.173:0/349197497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8080f5410 0x7ff808105710 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7ff7f8005010 tx=0x7ff7f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.770+0000 7ff7fdffb700 1 -- 172.21.15.173:0/349197497 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f8009320 con 0x7ff8080f5410 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.770+0000 7ff80f341700 1 -- 172.21.15.173:0/349197497 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff808105e00 con 0x7ff8080f5410 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.770+0000 7ff7fdffb700 1 -- 172.21.15.173:0/349197497 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7f8013380 con 0x7ff8080f5410 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.770+0000 7ff7fdffb700 1 -- 172.21.15.173:0/349197497 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f8014e50 con 0x7ff8080f5410 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.770+0000 7ff80f341700 1 -- 172.21.15.173:0/349197497 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff808106290 con 0x7ff8080f5410 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.771+0000 7ff7fdffb700 1 -- 172.21.15.173:0/349197497 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7f80263a0 con 0x7ff8080f5410 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.771+0000 7ff7fdffb700 1 --2- 172.21.15.173:0/349197497 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7f402ff10 0x7ff7f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.771+0000 7ff80f341700 1 -- 172.21.15.173:0/349197497 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff808045ad0 con 0x7ff8080f5410 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.771+0000 7ff7fdffb700 1 -- 172.21.15.173:0/349197497 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7f8040ff0 con 0x7ff8080f5410 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.771+0000 7ff80c8dc700 1 --2- 172.21.15.173:0/349197497 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7f402ff10 0x7ff7f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:49.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.772+0000 7ff80c8dc700 1 --2- 172.21.15.173:0/349197497 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7f402ff10 0x7ff7f40323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff8040096a0 tx=0x7ff804006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:49.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.776+0000 7ff7fdffb700 1 -- 172.21.15.173:0/349197497 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7f8026630 con 0x7ff8080f5410 2024-02-19T22:03:49.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.983+0000 7ff80f341700 1 -- 172.21.15.173:0/349197497 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8081070e0 con 0x7ff8080f5410 2024-02-19T22:03:49.983 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.984+0000 7ff7fdffb700 1 -- 172.21.15.173:0/349197497 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff7f8040020 con 0x7ff8080f5410 2024-02-19T22:03:49.984 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:49.984 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:49.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.985+0000 7ff80f341700 1 -- 172.21.15.173:0/349197497 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7f402ff10 msgr2=0x7ff7f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:49.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.985+0000 7ff80f341700 1 --2- 172.21.15.173:0/349197497 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7f402ff10 0x7ff7f40323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff8040096a0 tx=0x7ff804006b40).stop 2024-02-19T22:03:49.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.985+0000 7ff80f341700 1 -- 172.21.15.173:0/349197497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8080f5410 msgr2=0x7ff808105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:49.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.985+0000 7ff80f341700 1 --2- 172.21.15.173:0/349197497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8080f5410 0x7ff808105710 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7ff7f8005010 tx=0x7ff7f800b040).stop 2024-02-19T22:03:49.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.985+0000 7ff80f341700 1 -- 172.21.15.173:0/349197497 shutdown_connections 2024-02-19T22:03:49.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.985+0000 7ff80f341700 1 --2- 172.21.15.173:0/349197497 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff7f402ff10 0x7ff7f40323d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:49.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.985+0000 7ff80f341700 1 --2- 172.21.15.173:0/349197497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8080f5410 0x7ff808105710 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:49.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.985+0000 7ff80f341700 1 -- 172.21.15.173:0/349197497 >> 172.21.15.173:0/349197497 conn(0x7ff8080f0b70 msgr2=0x7ff8080f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:49.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.986+0000 7ff80f341700 1 -- 172.21.15.173:0/349197497 shutdown_connections 2024-02-19T22:03:49.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:49.986+0000 7ff80f341700 1 -- 172.21.15.173:0/349197497 wait complete. 2024-02-19T22:03:49.986 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:48.523703+0000 mgr.smithi143.bissbc (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:49.985015+0000 mon.smithi143 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.173:0/349197497' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:51.764 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:51.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:51.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:51 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:50.523929+0000 mgr.smithi143.bissbc (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:53.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.677+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/1677089388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd140f5410 msgr2=0x7fcd140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:53.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.677+0000 7fcd1b1c9700 1 --2- 172.21.15.173:0/1677089388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd140f5410 0x7fcd140f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fcd08009fb0 tx=0x7fcd08009b30).stop 2024-02-19T22:03:53.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.677+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/1677089388 shutdown_connections 2024-02-19T22:03:53.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.677+0000 7fcd1b1c9700 1 --2- 172.21.15.173:0/1677089388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd140f5410 0x7fcd140f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:53.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.677+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/1677089388 >> 172.21.15.173:0/1677089388 conn(0x7fcd140f0b70 msgr2=0x7fcd140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:53.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.677+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/1677089388 shutdown_connections 2024-02-19T22:03:53.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.677+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/1677089388 wait complete. 2024-02-19T22:03:53.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.677+0000 7fcd1b1c9700 1 Processor -- start 2024-02-19T22:03:53.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.678+0000 7fcd1b1c9700 1 -- start start 2024-02-19T22:03:53.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.678+0000 7fcd1b1c9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd140f5410 0x7fcd141094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:53.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.678+0000 7fcd1b1c9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd14060240 con 0x7fcd140f5410 2024-02-19T22:03:53.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.678+0000 7fcd18f65700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd140f5410 0x7fcd141094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:53.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.678+0000 7fcd18f65700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd140f5410 0x7fcd141094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:44902/0 (socket says 172.21.15.173:44902) 2024-02-19T22:03:53.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.678+0000 7fcd18f65700 1 -- 172.21.15.173:0/2368234500 learned_addr learned my addr 172.21.15.173:0/2368234500 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:53.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.678+0000 7fcd18f65700 1 -- 172.21.15.173:0/2368234500 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd08005040 con 0x7fcd140f5410 2024-02-19T22:03:53.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.678+0000 7fcd18f65700 1 --2- 172.21.15.173:0/2368234500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd140f5410 0x7fcd141094e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fcd08005010 tx=0x7fcd0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:53.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.679+0000 7fcd117fa700 1 -- 172.21.15.173:0/2368234500 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd08009320 con 0x7fcd140f5410 2024-02-19T22:03:53.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.679+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/2368234500 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd14060440 con 0x7fcd140f5410 2024-02-19T22:03:53.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.679+0000 7fcd117fa700 1 -- 172.21.15.173:0/2368234500 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd08013380 con 0x7fcd140f5410 2024-02-19T22:03:53.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.679+0000 7fcd117fa700 1 -- 172.21.15.173:0/2368234500 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd08014e50 con 0x7fcd140f5410 2024-02-19T22:03:53.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.679+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/2368234500 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd14109b90 con 0x7fcd140f5410 2024-02-19T22:03:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.679+0000 7fcd117fa700 1 -- 172.21.15.173:0/2368234500 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd08014420 con 0x7fcd140f5410 2024-02-19T22:03:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.680+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/2368234500 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd14045ad0 con 0x7fcd140f5410 2024-02-19T22:03:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.680+0000 7fcd117fa700 1 --2- 172.21.15.173:0/2368234500 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0402ff10 0x7fcd040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.680+0000 7fcd117fa700 1 -- 172.21.15.173:0/2368234500 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd08040e30 con 0x7fcd140f5410 2024-02-19T22:03:53.684 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.680+0000 7fcd13fff700 1 --2- 172.21.15.173:0/2368234500 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0402ff10 0x7fcd040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.680+0000 7fcd13fff700 1 --2- 172.21.15.173:0/2368234500 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0402ff10 0x7fcd040323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fcd000096a0 tx=0x7fcd00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:53.685 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.685+0000 7fcd117fa700 1 -- 172.21.15.173:0/2368234500 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd0801db70 con 0x7fcd140f5410 2024-02-19T22:03:53.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.890+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/2368234500 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd1410a3b0 con 0x7fcd140f5410 2024-02-19T22:03:53.890 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.891+0000 7fcd117fa700 1 -- 172.21.15.173:0/2368234500 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd0801db70 con 0x7fcd140f5410 2024-02-19T22:03:53.891 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:53.891 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:53.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.892+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/2368234500 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0402ff10 msgr2=0x7fcd040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:53.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.892+0000 7fcd1b1c9700 1 --2- 172.21.15.173:0/2368234500 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0402ff10 0x7fcd040323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fcd000096a0 tx=0x7fcd00006b40).stop 2024-02-19T22:03:53.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.892+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/2368234500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd140f5410 msgr2=0x7fcd141094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:53.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.892+0000 7fcd1b1c9700 1 --2- 172.21.15.173:0/2368234500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd140f5410 0x7fcd141094e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fcd08005010 tx=0x7fcd0800b040).stop 2024-02-19T22:03:53.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.893+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/2368234500 shutdown_connections 2024-02-19T22:03:53.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.893+0000 7fcd1b1c9700 1 --2- 172.21.15.173:0/2368234500 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fcd0402ff10 0x7fcd040323d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:53.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.893+0000 7fcd1b1c9700 1 --2- 172.21.15.173:0/2368234500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd140f5410 0x7fcd141094e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:53.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.893+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/2368234500 >> 172.21.15.173:0/2368234500 conn(0x7fcd140f0b70 msgr2=0x7fcd140f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:53.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.893+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/2368234500 shutdown_connections 2024-02-19T22:03:53.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:53.893+0000 7fcd1b1c9700 1 -- 172.21.15.173:0/2368234500 wait complete. 2024-02-19T22:03:53.893 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:52.524300+0000 mgr.smithi143.bissbc (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:53.892221+0000 mon.smithi143 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.173:0/2368234500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:55.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:55.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:54.524577+0000 mgr.smithi143.bissbc (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.659+0000 7f47b700c700 1 -- 172.21.15.173:0/1803608442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b00f6510 msgr2=0x7f47b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.659+0000 7f47b700c700 1 --2- 172.21.15.173:0/1803608442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b00f6510 0x7f47b00f6930 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f47a4009fb0 tx=0x7f47a4009b30).stop 2024-02-19T22:03:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.660+0000 7f47b700c700 1 -- 172.21.15.173:0/1803608442 shutdown_connections 2024-02-19T22:03:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.660+0000 7f47b700c700 1 --2- 172.21.15.173:0/1803608442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b00f6510 0x7f47b00f6930 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.660+0000 7f47b700c700 1 -- 172.21.15.173:0/1803608442 >> 172.21.15.173:0/1803608442 conn(0x7f47b00f1c70 msgr2=0x7f47b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.660+0000 7f47b700c700 1 -- 172.21.15.173:0/1803608442 shutdown_connections 2024-02-19T22:03:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.660+0000 7f47b700c700 1 -- 172.21.15.173:0/1803608442 wait complete. 2024-02-19T22:03:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.660+0000 7f47b700c700 1 Processor -- start 2024-02-19T22:03:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.660+0000 7f47b700c700 1 -- start start 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.660+0000 7f47b700c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b0108b00 0x7f47b0108f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.660+0000 7f47b700c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0109410 con 0x7f47b0108b00 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.661+0000 7f47b4da8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b0108b00 0x7f47b0108f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.661+0000 7f47b4da8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b0108b00 0x7f47b0108f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:44914/0 (socket says 172.21.15.173:44914) 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.661+0000 7f47b4da8700 1 -- 172.21.15.173:0/805389435 learned_addr learned my addr 172.21.15.173:0/805389435 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.661+0000 7f47b4da8700 1 -- 172.21.15.173:0/805389435 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a4005040 con 0x7f47b0108b00 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.661+0000 7f47b4da8700 1 --2- 172.21.15.173:0/805389435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b0108b00 0x7f47b0108f20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f47a4009f80 tx=0x7f47a4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.661+0000 7f47ad7fa700 1 -- 172.21.15.173:0/805389435 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a4013380 con 0x7f47b0108b00 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.661+0000 7f47b700c700 1 -- 172.21.15.173:0/805389435 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b0109610 con 0x7f47b0108b00 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.661+0000 7f47b700c700 1 -- 172.21.15.173:0/805389435 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b010a160 con 0x7f47b0108b00 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.661+0000 7f47ad7fa700 1 -- 172.21.15.173:0/805389435 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47a4013790 con 0x7f47b0108b00 2024-02-19T22:03:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.661+0000 7f47ad7fa700 1 -- 172.21.15.173:0/805389435 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a4014420 con 0x7f47b0108b00 2024-02-19T22:03:57.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.662+0000 7f47ad7fa700 1 -- 172.21.15.173:0/805389435 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47a4025060 con 0x7f47b0108b00 2024-02-19T22:03:57.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.662+0000 7f47ad7fa700 1 --2- 172.21.15.173:0/805389435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f47a002ffb0 0x7f47a0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:57.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.662+0000 7f47b700c700 1 -- 172.21.15.173:0/805389435 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b0059100 con 0x7f47b0108b00 2024-02-19T22:03:57.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.662+0000 7f47ad7fa700 1 -- 172.21.15.173:0/805389435 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47a4040ec0 con 0x7f47b0108b00 2024-02-19T22:03:57.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.663+0000 7f47affff700 1 --2- 172.21.15.173:0/805389435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f47a002ffb0 0x7f47a0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:57.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.663+0000 7f47affff700 1 --2- 172.21.15.173:0/805389435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f47a002ffb0 0x7f47a0032470 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f479c0096a0 tx=0x7f479c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:57.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.667+0000 7f47ad7fa700 1 -- 172.21.15.173:0/805389435 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47a4028440 con 0x7f47b0108b00 2024-02-19T22:03:57.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.874+0000 7f47b700c700 1 -- 172.21.15.173:0/805389435 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47b010a860 con 0x7f47b0108b00 2024-02-19T22:03:57.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.875+0000 7f47ad7fa700 1 -- 172.21.15.173:0/805389435 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f47a401d3a0 con 0x7f47b0108b00 2024-02-19T22:03:57.874 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:03:57.874 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:57.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.876+0000 7f47b700c700 1 -- 172.21.15.173:0/805389435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f47a002ffb0 msgr2=0x7f47a0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:57.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.876+0000 7f47b700c700 1 --2- 172.21.15.173:0/805389435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f47a002ffb0 0x7f47a0032470 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f479c0096a0 tx=0x7f479c006b40).stop 2024-02-19T22:03:57.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.876+0000 7f47b700c700 1 -- 172.21.15.173:0/805389435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b0108b00 msgr2=0x7f47b0108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:57.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.876+0000 7f47b700c700 1 --2- 172.21.15.173:0/805389435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b0108b00 0x7f47b0108f20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f47a4009f80 tx=0x7f47a4009b30).stop 2024-02-19T22:03:57.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.876+0000 7f47b700c700 1 -- 172.21.15.173:0/805389435 shutdown_connections 2024-02-19T22:03:57.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.876+0000 7f47b700c700 1 --2- 172.21.15.173:0/805389435 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f47a002ffb0 0x7f47a0032470 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:57.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.876+0000 7f47b700c700 1 --2- 172.21.15.173:0/805389435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47b0108b00 0x7f47b0108f20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:57.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.876+0000 7f47b700c700 1 -- 172.21.15.173:0/805389435 >> 172.21.15.173:0/805389435 conn(0x7f47b00f1c70 msgr2=0x7f47b00f26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:57.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.876+0000 7f47b700c700 1 -- 172.21.15.173:0/805389435 shutdown_connections 2024-02-19T22:03:57.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:03:57.876+0000 7f47b700c700 1 -- 172.21.15.173:0/805389435 wait complete. 2024-02-19T22:03:57.877 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:03:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:56.524924+0000 mgr.smithi143.bissbc (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:03:57.875840+0000 mon.smithi143 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.173:0/805389435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:59.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:59.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:59.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:03:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:03:58.525232+0000 mgr.smithi143.bissbc (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:01.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.639+0000 7f199ddcd700 1 -- 172.21.15.173:0/4201924387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1998060280 msgr2=0x7f19980f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:01.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.639+0000 7f199ddcd700 1 --2- 172.21.15.173:0/4201924387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1998060280 0x7f19980f90f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f1988009fb0 tx=0x7f1988009b30).stop 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.639+0000 7f199ddcd700 1 -- 172.21.15.173:0/4201924387 shutdown_connections 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.639+0000 7f199ddcd700 1 --2- 172.21.15.173:0/4201924387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1998060280 0x7f19980f90f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.639+0000 7f199ddcd700 1 -- 172.21.15.173:0/4201924387 >> 172.21.15.173:0/4201924387 conn(0x7f19980f0b70 msgr2=0x7f19980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.639+0000 7f199ddcd700 1 -- 172.21.15.173:0/4201924387 shutdown_connections 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.639+0000 7f199ddcd700 1 -- 172.21.15.173:0/4201924387 wait complete. 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.640+0000 7f199ddcd700 1 Processor -- start 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.640+0000 7f199ddcd700 1 -- start start 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.640+0000 7f199ddcd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1998060280 0x7f1998106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.640+0000 7f199ddcd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1998106c70 con 0x7f1998060280 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.641+0000 7f19977fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1998060280 0x7f1998106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.641+0000 7f19977fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1998060280 0x7f1998106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:51508/0 (socket says 172.21.15.173:51508) 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.641+0000 7f19977fe700 1 -- 172.21.15.173:0/2172150769 learned_addr learned my addr 172.21.15.173:0/2172150769 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:01.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.641+0000 7f19977fe700 1 -- 172.21.15.173:0/2172150769 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1988005040 con 0x7f1998060280 2024-02-19T22:04:01.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.641+0000 7f19977fe700 1 --2- 172.21.15.173:0/2172150769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1998060280 0x7f1998106780 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f19880089f0 tx=0x7f198800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.641+0000 7f197ffff700 1 -- 172.21.15.173:0/2172150769 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1988009320 con 0x7f1998060280 2024-02-19T22:04:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.641+0000 7f199ddcd700 1 -- 172.21.15.173:0/2172150769 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1998106e70 con 0x7f1998060280 2024-02-19T22:04:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.641+0000 7f197ffff700 1 -- 172.21.15.173:0/2172150769 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1988013380 con 0x7f1998060280 2024-02-19T22:04:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.641+0000 7f197ffff700 1 -- 172.21.15.173:0/2172150769 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1988014e20 con 0x7f1998060280 2024-02-19T22:04:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.641+0000 7f199ddcd700 1 -- 172.21.15.173:0/2172150769 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1998107300 con 0x7f1998060280 2024-02-19T22:04:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.642+0000 7f197ffff700 1 -- 172.21.15.173:0/2172150769 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1988014420 con 0x7f1998060280 2024-02-19T22:04:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.642+0000 7f199ddcd700 1 -- 172.21.15.173:0/2172150769 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1998045ad0 con 0x7f1998060280 2024-02-19T22:04:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.642+0000 7f197ffff700 1 --2- 172.21.15.173:0/2172150769 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f198002ff10 0x7f19800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.642+0000 7f197ffff700 1 -- 172.21.15.173:0/2172150769 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1988041010 con 0x7f1998060280 2024-02-19T22:04:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.642+0000 7f1996ffd700 1 --2- 172.21.15.173:0/2172150769 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f198002ff10 0x7f19800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.643+0000 7f1996ffd700 1 --2- 172.21.15.173:0/2172150769 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f198002ff10 0x7f19800323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f198c0096a0 tx=0x7f198c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:01.647 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.647+0000 7f197ffff700 1 -- 172.21.15.173:0/2172150769 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f198806d2b0 con 0x7f1998060280 2024-02-19T22:04:01.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.854+0000 7f199ddcd700 1 -- 172.21.15.173:0/2172150769 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19981081b0 con 0x7f1998060280 2024-02-19T22:04:01.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.855+0000 7f197ffff700 1 -- 172.21.15.173:0/2172150769 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f19880095f0 con 0x7f1998060280 2024-02-19T22:04:01.855 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:01.855 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:01.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.857+0000 7f199ddcd700 1 -- 172.21.15.173:0/2172150769 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f198002ff10 msgr2=0x7f19800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:01.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.857+0000 7f199ddcd700 1 --2- 172.21.15.173:0/2172150769 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f198002ff10 0x7f19800323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f198c0096a0 tx=0x7f198c006b40).stop 2024-02-19T22:04:01.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.857+0000 7f199ddcd700 1 -- 172.21.15.173:0/2172150769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1998060280 msgr2=0x7f1998106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:01.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.857+0000 7f199ddcd700 1 --2- 172.21.15.173:0/2172150769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1998060280 0x7f1998106780 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f19880089f0 tx=0x7f198800b040).stop 2024-02-19T22:04:01.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.857+0000 7f199ddcd700 1 -- 172.21.15.173:0/2172150769 shutdown_connections 2024-02-19T22:04:01.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.857+0000 7f199ddcd700 1 --2- 172.21.15.173:0/2172150769 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f198002ff10 0x7f19800323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:01.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.857+0000 7f199ddcd700 1 --2- 172.21.15.173:0/2172150769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1998060280 0x7f1998106780 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:01.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.857+0000 7f199ddcd700 1 -- 172.21.15.173:0/2172150769 >> 172.21.15.173:0/2172150769 conn(0x7f19980f0b70 msgr2=0x7f19980f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:01.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.857+0000 7f199ddcd700 1 -- 172.21.15.173:0/2172150769 shutdown_connections 2024-02-19T22:04:01.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:01.857+0000 7f199ddcd700 1 -- 172.21.15.173:0/2172150769 wait complete. 2024-02-19T22:04:01.857 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:01.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:01 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:00.525579+0000 mgr.smithi143.bissbc (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:01.856247+0000 mon.smithi143 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.173:0/2172150769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:03.665 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:03.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:02.525917+0000 mgr.smithi143.bissbc (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.666+0000 7f27d359e700 1 -- 172.21.15.173:0/1684282681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27d40601e0 msgr2=0x7f27d40f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.666+0000 7f27d359e700 1 --2- 172.21.15.173:0/1684282681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27d40601e0 0x7f27d40f8f20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f27c4009fb0 tx=0x7f27c4009b30).stop 2024-02-19T22:04:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.666+0000 7f27d359e700 1 -- 172.21.15.173:0/1684282681 shutdown_connections 2024-02-19T22:04:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.666+0000 7f27d359e700 1 --2- 172.21.15.173:0/1684282681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27d40601e0 0x7f27d40f8f20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.666+0000 7f27d359e700 1 -- 172.21.15.173:0/1684282681 >> 172.21.15.173:0/1684282681 conn(0x7f27d40f09a0 msgr2=0x7f27d40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.666+0000 7f27d359e700 1 -- 172.21.15.173:0/1684282681 shutdown_connections 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.666+0000 7f27d359e700 1 -- 172.21.15.173:0/1684282681 wait complete. 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.666+0000 7f27d359e700 1 Processor -- start 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.666+0000 7f27d359e700 1 -- start start 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.666+0000 7f27d359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27d40601e0 0x7f27d4106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.666+0000 7f27d359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4106af0 con 0x7f27d40601e0 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.667+0000 7f27d259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27d40601e0 0x7f27d4106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.667+0000 7f27d259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27d40601e0 0x7f27d4106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:51512/0 (socket says 172.21.15.173:51512) 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.667+0000 7f27d259c700 1 -- 172.21.15.173:0/2815109338 learned_addr learned my addr 172.21.15.173:0/2815109338 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.667+0000 7f27d259c700 1 -- 172.21.15.173:0/2815109338 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c4005040 con 0x7f27d40601e0 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.667+0000 7f27d259c700 1 --2- 172.21.15.173:0/2815109338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27d40601e0 0x7f27d4106600 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f27c4005010 tx=0x7f27c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.667+0000 7f27c2ffd700 1 -- 172.21.15.173:0/2815109338 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c4009320 con 0x7f27d40601e0 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.667+0000 7f27d359e700 1 -- 172.21.15.173:0/2815109338 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d4106cf0 con 0x7f27d40601e0 2024-02-19T22:04:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.667+0000 7f27c2ffd700 1 -- 172.21.15.173:0/2815109338 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27c4013380 con 0x7f27d40601e0 2024-02-19T22:04:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.667+0000 7f27c2ffd700 1 -- 172.21.15.173:0/2815109338 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c4014e50 con 0x7f27d40601e0 2024-02-19T22:04:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.667+0000 7f27d359e700 1 -- 172.21.15.173:0/2815109338 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d4107180 con 0x7f27d40601e0 2024-02-19T22:04:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.668+0000 7f27c2ffd700 1 -- 172.21.15.173:0/2815109338 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27c40095f0 con 0x7f27d40601e0 2024-02-19T22:04:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.668+0000 7f27d359e700 1 -- 172.21.15.173:0/2815109338 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d4045ad0 con 0x7f27d40601e0 2024-02-19T22:04:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.668+0000 7f27c2ffd700 1 --2- 172.21.15.173:0/2815109338 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f27bc02ff10 0x7f27bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.668+0000 7f27c2ffd700 1 -- 172.21.15.173:0/2815109338 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27c4040fa0 con 0x7f27d40601e0 2024-02-19T22:04:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.668+0000 7f27d1d9b700 1 --2- 172.21.15.173:0/2815109338 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f27bc02ff10 0x7f27bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.669+0000 7f27d1d9b700 1 --2- 172.21.15.173:0/2815109338 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f27bc02ff10 0x7f27bc0323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f27c80096a0 tx=0x7f27c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.673+0000 7f27c2ffd700 1 -- 172.21.15.173:0/2815109338 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27c4014420 con 0x7f27d40601e0 2024-02-19T22:04:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.879+0000 7f27d359e700 1 -- 172.21.15.173:0/2815109338 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27d4108030 con 0x7f27d40601e0 2024-02-19T22:04:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.880+0000 7f27c2ffd700 1 -- 172.21.15.173:0/2815109338 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f27c4010230 con 0x7f27d40601e0 2024-02-19T22:04:05.880 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:05.880 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:05.881 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.882+0000 7f27d359e700 1 -- 172.21.15.173:0/2815109338 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f27bc02ff10 msgr2=0x7f27bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:05.881 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.882+0000 7f27d359e700 1 --2- 172.21.15.173:0/2815109338 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f27bc02ff10 0x7f27bc0323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f27c80096a0 tx=0x7f27c8006b40).stop 2024-02-19T22:04:05.881 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.882+0000 7f27d359e700 1 -- 172.21.15.173:0/2815109338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27d40601e0 msgr2=0x7f27d4106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:05.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.882+0000 7f27d359e700 1 --2- 172.21.15.173:0/2815109338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27d40601e0 0x7f27d4106600 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f27c4005010 tx=0x7f27c400b040).stop 2024-02-19T22:04:05.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.882+0000 7f27d359e700 1 -- 172.21.15.173:0/2815109338 shutdown_connections 2024-02-19T22:04:05.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.882+0000 7f27d359e700 1 --2- 172.21.15.173:0/2815109338 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f27bc02ff10 0x7f27bc0323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:05.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.882+0000 7f27d359e700 1 --2- 172.21.15.173:0/2815109338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27d40601e0 0x7f27d4106600 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:05.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.882+0000 7f27d359e700 1 -- 172.21.15.173:0/2815109338 >> 172.21.15.173:0/2815109338 conn(0x7f27d40f09a0 msgr2=0x7f27d40f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:05.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.882+0000 7f27d359e700 1 -- 172.21.15.173:0/2815109338 shutdown_connections 2024-02-19T22:04:05.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:05.882+0000 7f27d359e700 1 -- 172.21.15.173:0/2815109338 wait complete. 2024-02-19T22:04:05.882 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:04.526196+0000 mgr.smithi143.bissbc (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:05.881131+0000 mon.smithi143 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.173:0/2815109338' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:07.764 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:07.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:06.526563+0000 mgr.smithi143.bissbc (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:09.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.781+0000 7fd57fa68700 1 -- 172.21.15.173:0/795885339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5780f47c0 msgr2=0x7fd5780f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.781+0000 7fd57fa68700 1 --2- 172.21.15.173:0/795885339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5780f47c0 0x7fd5780f4be0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fd568009960 tx=0x7fd568004d10).stop 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.781+0000 7fd57fa68700 1 -- 172.21.15.173:0/795885339 shutdown_connections 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.781+0000 7fd57fa68700 1 --2- 172.21.15.173:0/795885339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5780f47c0 0x7fd5780f4be0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.781+0000 7fd57fa68700 1 -- 172.21.15.173:0/795885339 >> 172.21.15.173:0/795885339 conn(0x7fd5780f02e0 msgr2=0x7fd5780f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.782+0000 7fd57fa68700 1 -- 172.21.15.173:0/795885339 shutdown_connections 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.782+0000 7fd57fa68700 1 -- 172.21.15.173:0/795885339 wait complete. 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.782+0000 7fd57fa68700 1 Processor -- start 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.782+0000 7fd57fa68700 1 -- start start 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.782+0000 7fd57fa68700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5780f47c0 0x7fd5781067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.782+0000 7fd57fa68700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd578106cc0 con 0x7fd5780f47c0 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.782+0000 7fd57d804700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5780f47c0 0x7fd5781067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.782+0000 7fd57d804700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5780f47c0 0x7fd5781067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:51530/0 (socket says 172.21.15.173:51530) 2024-02-19T22:04:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.782+0000 7fd57d804700 1 -- 172.21.15.173:0/3070694817 learned_addr learned my addr 172.21.15.173:0/3070694817 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.783+0000 7fd57d804700 1 -- 172.21.15.173:0/3070694817 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd568009640 con 0x7fd5780f47c0 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.783+0000 7fd57d804700 1 --2- 172.21.15.173:0/3070694817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5780f47c0 0x7fd5781067d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fd568004b70 tx=0x7fd568004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.783+0000 7fd5767fc700 1 -- 172.21.15.173:0/3070694817 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd568018070 con 0x7fd5780f47c0 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.783+0000 7fd5767fc700 1 -- 172.21.15.173:0/3070694817 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd56800b970 con 0x7fd5780f47c0 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.783+0000 7fd57fa68700 1 -- 172.21.15.173:0/3070694817 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd578106ec0 con 0x7fd5780f47c0 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.783+0000 7fd5767fc700 1 -- 172.21.15.173:0/3070694817 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd56801c560 con 0x7fd5780f47c0 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.783+0000 7fd57fa68700 1 -- 172.21.15.173:0/3070694817 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd578107350 con 0x7fd5780f47c0 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.784+0000 7fd5767fc700 1 -- 172.21.15.173:0/3070694817 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd568013630 con 0x7fd5780f47c0 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.784+0000 7fd57fa68700 1 -- 172.21.15.173:0/3070694817 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd578045ad0 con 0x7fd5780f47c0 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.784+0000 7fd5767fc700 1 --2- 172.21.15.173:0/3070694817 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd56002ffb0 0x7fd560032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.784+0000 7fd5767fc700 1 -- 172.21.15.173:0/3070694817 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd568041220 con 0x7fd5780f47c0 2024-02-19T22:04:09.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.784+0000 7fd57d003700 1 --2- 172.21.15.173:0/3070694817 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd56002ffb0 0x7fd560032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:09.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.784+0000 7fd57d003700 1 --2- 172.21.15.173:0/3070694817 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd56002ffb0 0x7fd560032470 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fd56c0096a0 tx=0x7fd56c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:09.793 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.789+0000 7fd5767fc700 1 -- 172.21.15.173:0/3070694817 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd568013a30 con 0x7fd5780f47c0 2024-02-19T22:04:09.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:09 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:08.526909+0000 mgr.smithi143.bissbc (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:09.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.997+0000 7fd57fa68700 1 -- 172.21.15.173:0/3070694817 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd578108200 con 0x7fd5780f47c0 2024-02-19T22:04:09.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.997+0000 7fd5767fc700 1 -- 172.21.15.173:0/3070694817 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd568012380 con 0x7fd5780f47c0 2024-02-19T22:04:09.997 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:09.997 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:10.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.999+0000 7fd57fa68700 1 -- 172.21.15.173:0/3070694817 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd56002ffb0 msgr2=0x7fd560032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:10.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.999+0000 7fd57fa68700 1 --2- 172.21.15.173:0/3070694817 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd56002ffb0 0x7fd560032470 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fd56c0096a0 tx=0x7fd56c006b40).stop 2024-02-19T22:04:10.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.999+0000 7fd57fa68700 1 -- 172.21.15.173:0/3070694817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5780f47c0 msgr2=0x7fd5781067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:10.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.999+0000 7fd57fa68700 1 --2- 172.21.15.173:0/3070694817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5780f47c0 0x7fd5781067d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fd568004b70 tx=0x7fd568004d10).stop 2024-02-19T22:04:10.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.999+0000 7fd57fa68700 1 -- 172.21.15.173:0/3070694817 shutdown_connections 2024-02-19T22:04:10.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.999+0000 7fd57fa68700 1 --2- 172.21.15.173:0/3070694817 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fd56002ffb0 0x7fd560032470 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:10.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.999+0000 7fd57fa68700 1 --2- 172.21.15.173:0/3070694817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5780f47c0 0x7fd5781067d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:10.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.999+0000 7fd57fa68700 1 -- 172.21.15.173:0/3070694817 >> 172.21.15.173:0/3070694817 conn(0x7fd5780f02e0 msgr2=0x7fd5780f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:10.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:09.999+0000 7fd57fa68700 1 -- 172.21.15.173:0/3070694817 shutdown_connections 2024-02-19T22:04:10.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:10.000+0000 7fd57fa68700 1 -- 172.21.15.173:0/3070694817 wait complete. 2024-02-19T22:04:10.000 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:09.998630+0000 mon.smithi143 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.173:0/3070694817' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:11.827 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:11.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:12.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:10.527321+0000 mgr.smithi143.bissbc (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:13.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.875+0000 7f7994563700 1 -- 172.21.15.173:0/1946556085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f798c0f5410 msgr2=0x7f798c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:13.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.875+0000 7f7994563700 1 --2- 172.21.15.173:0/1946556085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f798c0f5410 0x7f798c0f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f797c009fb0 tx=0x7f797c009b30).stop 2024-02-19T22:04:13.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.875+0000 7f7994563700 1 -- 172.21.15.173:0/1946556085 shutdown_connections 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.875+0000 7f7994563700 1 --2- 172.21.15.173:0/1946556085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f798c0f5410 0x7f798c0f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.875+0000 7f7994563700 1 -- 172.21.15.173:0/1946556085 >> 172.21.15.173:0/1946556085 conn(0x7f798c0f0b70 msgr2=0x7f798c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.875+0000 7f7994563700 1 -- 172.21.15.173:0/1946556085 shutdown_connections 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.875+0000 7f7994563700 1 -- 172.21.15.173:0/1946556085 wait complete. 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.876+0000 7f7994563700 1 Processor -- start 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.876+0000 7f7994563700 1 -- start start 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.876+0000 7f7994563700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f798c0f5410 0x7f798c106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.876+0000 7f7994563700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c106cf0 con 0x7f798c0f5410 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.876+0000 7f79922ff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f798c0f5410 0x7f798c106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.876+0000 7f79922ff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f798c0f5410 0x7f798c106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43620/0 (socket says 172.21.15.173:43620) 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.876+0000 7f79922ff700 1 -- 172.21.15.173:0/590549284 learned_addr learned my addr 172.21.15.173:0/590549284 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.876+0000 7f79922ff700 1 -- 172.21.15.173:0/590549284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c005040 con 0x7f798c0f5410 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.876+0000 7f79922ff700 1 --2- 172.21.15.173:0/590549284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f798c0f5410 0x7f798c106800 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f797c005010 tx=0x7f797c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:13.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.877+0000 7f7982ffd700 1 -- 172.21.15.173:0/590549284 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f797c009320 con 0x7f798c0f5410 2024-02-19T22:04:13.878 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.877+0000 7f7994563700 1 -- 172.21.15.173:0/590549284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798c106ef0 con 0x7f798c0f5410 2024-02-19T22:04:13.878 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.877+0000 7f7982ffd700 1 -- 172.21.15.173:0/590549284 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f797c013380 con 0x7f798c0f5410 2024-02-19T22:04:13.878 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.877+0000 7f7982ffd700 1 -- 172.21.15.173:0/590549284 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f797c014e50 con 0x7f798c0f5410 2024-02-19T22:04:13.878 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.877+0000 7f7994563700 1 -- 172.21.15.173:0/590549284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798c107380 con 0x7f798c0f5410 2024-02-19T22:04:13.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.877+0000 7f7982ffd700 1 -- 172.21.15.173:0/590549284 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f797c014420 con 0x7f798c0f5410 2024-02-19T22:04:13.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.877+0000 7f7994563700 1 -- 172.21.15.173:0/590549284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798c045ad0 con 0x7f798c0f5410 2024-02-19T22:04:13.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.878+0000 7f7982ffd700 1 --2- 172.21.15.173:0/590549284 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f797802ff10 0x7f79780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:13.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.878+0000 7f7982ffd700 1 -- 172.21.15.173:0/590549284 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f797c040f40 con 0x7f798c0f5410 2024-02-19T22:04:13.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.878+0000 7f7991afe700 1 --2- 172.21.15.173:0/590549284 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f797802ff10 0x7f79780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:13.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.878+0000 7f7991afe700 1 --2- 172.21.15.173:0/590549284 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f797802ff10 0x7f79780323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f79880096a0 tx=0x7f7988006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:13.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:13.882+0000 7f7982ffd700 1 -- 172.21.15.173:0/590549284 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f797c0095f0 con 0x7f798c0f5410 2024-02-19T22:04:14.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.091+0000 7f7994563700 1 -- 172.21.15.173:0/590549284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f798c108230 con 0x7f798c0f5410 2024-02-19T22:04:14.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.092+0000 7f7982ffd700 1 -- 172.21.15.173:0/590549284 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f797c01db70 con 0x7f798c0f5410 2024-02-19T22:04:14.091 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:14.091 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:14.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.093+0000 7f7994563700 1 -- 172.21.15.173:0/590549284 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f797802ff10 msgr2=0x7f79780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:14.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.093+0000 7f7994563700 1 --2- 172.21.15.173:0/590549284 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f797802ff10 0x7f79780323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f79880096a0 tx=0x7f7988006b40).stop 2024-02-19T22:04:14.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.093+0000 7f7994563700 1 -- 172.21.15.173:0/590549284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f798c0f5410 msgr2=0x7f798c106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:14.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.093+0000 7f7994563700 1 --2- 172.21.15.173:0/590549284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f798c0f5410 0x7f798c106800 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f797c005010 tx=0x7f797c00b040).stop 2024-02-19T22:04:14.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.093+0000 7f7994563700 1 -- 172.21.15.173:0/590549284 shutdown_connections 2024-02-19T22:04:14.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.093+0000 7f7994563700 1 --2- 172.21.15.173:0/590549284 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f797802ff10 0x7f79780323d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:14.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.093+0000 7f7994563700 1 --2- 172.21.15.173:0/590549284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f798c0f5410 0x7f798c106800 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:14.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.093+0000 7f7994563700 1 -- 172.21.15.173:0/590549284 >> 172.21.15.173:0/590549284 conn(0x7f798c0f0b70 msgr2=0x7f798c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:14.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.094+0000 7f7994563700 1 -- 172.21.15.173:0/590549284 shutdown_connections 2024-02-19T22:04:14.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:14.094+0000 7f7994563700 1 -- 172.21.15.173:0/590549284 wait complete. 2024-02-19T22:04:14.094 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:12.527600+0000 mgr.smithi143.bissbc (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:14.092916+0000 mon.smithi143 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.173:0/590549284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:15.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:15.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:14.527847+0000 mgr.smithi143.bissbc (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:17.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.689+0000 7f420c836700 1 -- 172.21.15.173:0/4093078451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4208060220 msgr2=0x7f42080f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:17.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.689+0000 7f420c836700 1 --2- 172.21.15.173:0/4093078451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4208060220 0x7f42080f9150 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f41f0009fb0 tx=0x7f41f0009b30).stop 2024-02-19T22:04:17.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.689+0000 7f420c836700 1 -- 172.21.15.173:0/4093078451 shutdown_connections 2024-02-19T22:04:17.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.689+0000 7f420c836700 1 --2- 172.21.15.173:0/4093078451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4208060220 0x7f42080f9150 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:17.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.689+0000 7f420c836700 1 -- 172.21.15.173:0/4093078451 >> 172.21.15.173:0/4093078451 conn(0x7f42080f0b70 msgr2=0x7f42080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:17.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.690+0000 7f420c836700 1 -- 172.21.15.173:0/4093078451 shutdown_connections 2024-02-19T22:04:17.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.690+0000 7f420c836700 1 -- 172.21.15.173:0/4093078451 wait complete. 2024-02-19T22:04:17.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.690+0000 7f420c836700 1 Processor -- start 2024-02-19T22:04:17.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.690+0000 7f420c836700 1 -- start start 2024-02-19T22:04:17.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.690+0000 7f420c836700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4208060220 0x7f42080f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:17.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.690+0000 7f420c836700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42080f5050 con 0x7f4208060220 2024-02-19T22:04:17.692 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.690+0000 7f420659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4208060220 0x7f42080f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.690+0000 7f420659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4208060220 0x7f42080f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:43646/0 (socket says 172.21.15.173:43646) 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.691+0000 7f420659c700 1 -- 172.21.15.173:0/1439111806 learned_addr learned my addr 172.21.15.173:0/1439111806 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.691+0000 7f420659c700 1 -- 172.21.15.173:0/1439111806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f0005040 con 0x7f4208060220 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.691+0000 7f420659c700 1 --2- 172.21.15.173:0/1439111806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4208060220 0x7f42080f6bd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f41f00089f0 tx=0x7f41f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.691+0000 7f41feffd700 1 -- 172.21.15.173:0/1439111806 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41f0009320 con 0x7f4208060220 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.691+0000 7f420c836700 1 -- 172.21.15.173:0/1439111806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42080f51f0 con 0x7f4208060220 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.691+0000 7f41feffd700 1 -- 172.21.15.173:0/1439111806 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41f0013380 con 0x7f4208060220 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.691+0000 7f41feffd700 1 -- 172.21.15.173:0/1439111806 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41f0014e20 con 0x7f4208060220 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.691+0000 7f420c836700 1 -- 172.21.15.173:0/1439111806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42080f5680 con 0x7f4208060220 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.692+0000 7f41feffd700 1 -- 172.21.15.173:0/1439111806 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41f00095f0 con 0x7f4208060220 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.692+0000 7f420c836700 1 -- 172.21.15.173:0/1439111806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4208059100 con 0x7f4208060220 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.692+0000 7f41feffd700 1 --2- 172.21.15.173:0/1439111806 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41f402ff60 0x7f41f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.692+0000 7f41feffd700 1 -- 172.21.15.173:0/1439111806 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41f0040f00 con 0x7f4208060220 2024-02-19T22:04:17.693 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.692+0000 7f4205d9b700 1 --2- 172.21.15.173:0/1439111806 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41f402ff60 0x7f41f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:17.694 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.693+0000 7f4205d9b700 1 --2- 172.21.15.173:0/1439111806 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41f402ff60 0x7f41f4032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f41f80096a0 tx=0x7f41f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:17.696 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.697+0000 7f41feffd700 1 -- 172.21.15.173:0/1439111806 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41f0014420 con 0x7f4208060220 2024-02-19T22:04:17.898 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.899+0000 7f420c836700 1 -- 172.21.15.173:0/1439111806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42080fc090 con 0x7f4208060220 2024-02-19T22:04:17.899 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.899+0000 7f41feffd700 1 -- 172.21.15.173:0/1439111806 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f41f0010230 con 0x7f4208060220 2024-02-19T22:04:17.899 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:17.899 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:17.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.901+0000 7f420c836700 1 -- 172.21.15.173:0/1439111806 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41f402ff60 msgr2=0x7f41f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:17.900 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.901+0000 7f420c836700 1 --2- 172.21.15.173:0/1439111806 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41f402ff60 0x7f41f4032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f41f80096a0 tx=0x7f41f8006b40).stop 2024-02-19T22:04:17.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.901+0000 7f420c836700 1 -- 172.21.15.173:0/1439111806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4208060220 msgr2=0x7f42080f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:17.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.901+0000 7f420c836700 1 --2- 172.21.15.173:0/1439111806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4208060220 0x7f42080f6bd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f41f00089f0 tx=0x7f41f000b040).stop 2024-02-19T22:04:17.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.901+0000 7f420c836700 1 -- 172.21.15.173:0/1439111806 shutdown_connections 2024-02-19T22:04:17.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.901+0000 7f420c836700 1 --2- 172.21.15.173:0/1439111806 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f41f402ff60 0x7f41f4032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:17.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.901+0000 7f420c836700 1 --2- 172.21.15.173:0/1439111806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4208060220 0x7f42080f6bd0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:17.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.901+0000 7f420c836700 1 -- 172.21.15.173:0/1439111806 >> 172.21.15.173:0/1439111806 conn(0x7f42080f0b70 msgr2=0x7f42080f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:17.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.901+0000 7f420c836700 1 -- 172.21.15.173:0/1439111806 shutdown_connections 2024-02-19T22:04:17.901 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:17.901+0000 7f420c836700 1 -- 172.21.15.173:0/1439111806 wait complete. 2024-02-19T22:04:17.901 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:16.528052+0000 mgr.smithi143.bissbc (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:17.900683+0000 mon.smithi143 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.173:0/1439111806' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:19.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:19.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:18.528313+0000 mgr.smithi143.bissbc (mgr.14162) 307 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:21.754 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.754+0000 7f9438922700 1 -- 172.21.15.173:0/2412509655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94340f5410 msgr2=0x7f94340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:21.754 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.754+0000 7f9438922700 1 --2- 172.21.15.173:0/2412509655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94340f5410 0x7f94340f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f9428009fb0 tx=0x7f9428009b30).stop 2024-02-19T22:04:21.754 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.755+0000 7f9438922700 1 -- 172.21.15.173:0/2412509655 shutdown_connections 2024-02-19T22:04:21.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.755+0000 7f9438922700 1 --2- 172.21.15.173:0/2412509655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94340f5410 0x7f94340f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:21.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.755+0000 7f9438922700 1 -- 172.21.15.173:0/2412509655 >> 172.21.15.173:0/2412509655 conn(0x7f94340f0b70 msgr2=0x7f94340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:21.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.755+0000 7f9438922700 1 -- 172.21.15.173:0/2412509655 shutdown_connections 2024-02-19T22:04:21.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.755+0000 7f9438922700 1 -- 172.21.15.173:0/2412509655 wait complete. 2024-02-19T22:04:21.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.755+0000 7f9438922700 1 Processor -- start 2024-02-19T22:04:21.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.755+0000 7f9438922700 1 -- start start 2024-02-19T22:04:21.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.755+0000 7f9438922700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94340f5410 0x7f9434105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:21.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.755+0000 7f9438922700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9434105c10 con 0x7f94340f5410 2024-02-19T22:04:21.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.756+0000 7f943259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94340f5410 0x7f9434105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:21.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.756+0000 7f943259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94340f5410 0x7f9434105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:52890/0 (socket says 172.21.15.173:52890) 2024-02-19T22:04:21.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.756+0000 7f943259c700 1 -- 172.21.15.173:0/305978128 learned_addr learned my addr 172.21.15.173:0/305978128 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.756+0000 7f943259c700 1 -- 172.21.15.173:0/305978128 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9428005040 con 0x7f94340f5410 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.756+0000 7f943259c700 1 --2- 172.21.15.173:0/305978128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94340f5410 0x7f9434105720 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f94280089f0 tx=0x7f942800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.756+0000 7f9422ffd700 1 -- 172.21.15.173:0/305978128 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9428009320 con 0x7f94340f5410 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.756+0000 7f9422ffd700 1 -- 172.21.15.173:0/305978128 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9428013380 con 0x7f94340f5410 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.756+0000 7f9422ffd700 1 -- 172.21.15.173:0/305978128 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f942801ed80 con 0x7f94340f5410 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.756+0000 7f9438922700 1 -- 172.21.15.173:0/305978128 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9434105e10 con 0x7f94340f5410 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.756+0000 7f9438922700 1 -- 172.21.15.173:0/305978128 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94341062a0 con 0x7f94340f5410 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.757+0000 7f9422ffd700 1 -- 172.21.15.173:0/305978128 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f942801e420 con 0x7f94340f5410 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.757+0000 7f9438922700 1 -- 172.21.15.173:0/305978128 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9434045ad0 con 0x7f94340f5410 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.757+0000 7f9422ffd700 1 --2- 172.21.15.173:0/305978128 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f941c02ff60 0x7f941c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.757+0000 7f9422ffd700 1 -- 172.21.15.173:0/305978128 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9428040f80 con 0x7f94340f5410 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.757+0000 7f9431d9b700 1 --2- 172.21.15.173:0/305978128 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f941c02ff60 0x7f941c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:21.758 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.758+0000 7f9431d9b700 1 --2- 172.21.15.173:0/305978128 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f941c02ff60 0x7f941c032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f94240096a0 tx=0x7f9424006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:21.762 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.762+0000 7f9422ffd700 1 -- 172.21.15.173:0/305978128 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f942806d2b0 con 0x7f94340f5410 2024-02-19T22:04:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:20.572746+0000 mon.smithi143 (mon.0) 288 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T22:04:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:20.572948+0000 mon.smithi143 (mon.0) 289 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:04:21.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.966+0000 7f9438922700 1 -- 172.21.15.173:0/305978128 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9434107150 con 0x7f94340f5410 2024-02-19T22:04:21.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.967+0000 7f9422ffd700 1 -- 172.21.15.173:0/305978128 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f94280095f0 con 0x7f94340f5410 2024-02-19T22:04:21.967 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:21.967 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:21.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.968+0000 7f9438922700 1 -- 172.21.15.173:0/305978128 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f941c02ff60 msgr2=0x7f941c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:21.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.968+0000 7f9438922700 1 --2- 172.21.15.173:0/305978128 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f941c02ff60 0x7f941c032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f94240096a0 tx=0x7f9424006b40).stop 2024-02-19T22:04:21.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.968+0000 7f9438922700 1 -- 172.21.15.173:0/305978128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94340f5410 msgr2=0x7f9434105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:21.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.968+0000 7f9438922700 1 --2- 172.21.15.173:0/305978128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94340f5410 0x7f9434105720 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f94280089f0 tx=0x7f942800b040).stop 2024-02-19T22:04:21.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.968+0000 7f9438922700 1 -- 172.21.15.173:0/305978128 shutdown_connections 2024-02-19T22:04:21.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.968+0000 7f9438922700 1 --2- 172.21.15.173:0/305978128 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f941c02ff60 0x7f941c032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:21.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.968+0000 7f9438922700 1 --2- 172.21.15.173:0/305978128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94340f5410 0x7f9434105720 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:21.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.968+0000 7f9438922700 1 -- 172.21.15.173:0/305978128 >> 172.21.15.173:0/305978128 conn(0x7f94340f0b70 msgr2=0x7f94340f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:21.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.968+0000 7f9438922700 1 -- 172.21.15.173:0/305978128 shutdown_connections 2024-02-19T22:04:21.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:21.968+0000 7f9438922700 1 -- 172.21.15.173:0/305978128 wait complete. 2024-02-19T22:04:21.970 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:20.528526+0000 mgr.smithi143.bissbc (mgr.14162) 308 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:21.968057+0000 mon.smithi143 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.173:0/305978128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:22.462041+0000 mon.smithi143 (mon.0) 291 : audit [DBG] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:04:23.780 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:23.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:23.855 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T22:04:22.461443+0000 mgr.smithi143.bissbc (mgr.14162) 309 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi143 ... 2024-02-19T22:04:23.855 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:04:23.856 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:04:23.856 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T22:02:29Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:04:23.856 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T22:03:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:04:23.856 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T22:03:45Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:04:23.856 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:04:23.856 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Traceback (most recent call last): 2024-02-19T22:04:23.857 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:04:23.857 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: main() 2024-02-19T22:04:23.857 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:04:23.857 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: r = ctx.func(ctx) 2024-02-19T22:04:23.857 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:04:23.857 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: return func(ctx) 2024-02-19T22:04:23.858 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:04:23.858 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:04:23.858 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:04:23.858 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:04:23.859 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:04:23.859 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: raise RuntimeError('uid/gid not found') 2024-02-19T22:04:23.859 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: RuntimeError: uid/gid not found 2024-02-19T22:04:23.859 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Traceback (most recent call last): 2024-02-19T22:04:23.859 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:04:23.859 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: yield (conn, connr) 2024-02-19T22:04:23.860 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:04:23.860 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: code, '\n'.join(err))) 2024-02-19T22:04:23.860 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi143 ... 2024-02-19T22:04:23.860 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi143 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:04:23.860 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:04:23.860 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T22:02:29Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:04:23.861 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T22:03:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:04:23.861 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr time="2024-02-19T22:03:45Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:04:23.861 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:04:23.861 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: Traceback (most recent call last): 2024-02-19T22:04:23.861 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:04:23.861 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: main() 2024-02-19T22:04:23.862 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:04:23.862 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: r = ctx.func(ctx) 2024-02-19T22:04:23.862 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:04:23.864 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: return func(ctx) 2024-02-19T22:04:23.864 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:04:23.864 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:04:23.864 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:04:23.864 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:04:23.865 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: File "/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:04:23.865 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: raise RuntimeError('uid/gid not found') 2024-02-19T22:04:23.865 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: RuntimeError: uid/gid not found 2024-02-19T22:04:23.865 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:22.823046+0000 mon.smithi143 (mon.0) 292 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T22:04:23.865 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:23.052589+0000 mon.smithi143 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T22:04:24.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:22.528789+0000 mgr.smithi143.bissbc (mgr.14162) 310 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:24.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:24 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:24.405602+0000 mon.smithi143 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T22:04:26.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.325+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1964501455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b280f6510 msgr2=0x7f9b280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.325+0000 7f9b2d4f6700 1 --2- 172.21.15.173:0/1964501455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b280f6510 0x7f9b280f6930 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f9b1c009fb0 tx=0x7f9b1c009b30).stop 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.325+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1964501455 shutdown_connections 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.325+0000 7f9b2d4f6700 1 --2- 172.21.15.173:0/1964501455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b280f6510 0x7f9b280f6930 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.325+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1964501455 >> 172.21.15.173:0/1964501455 conn(0x7f9b280f1c70 msgr2=0x7f9b280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.325+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1964501455 shutdown_connections 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.325+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1964501455 wait complete. 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.326+0000 7f9b2d4f6700 1 Processor -- start 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.326+0000 7f9b2d4f6700 1 -- start start 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.326+0000 7f9b2d4f6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b280f6510 0x7f9b281067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.326+0000 7f9b2d4f6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b28106cd0 con 0x7f9b280f6510 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.326+0000 7f9b26ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b280f6510 0x7f9b281067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.326+0000 7f9b26ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b280f6510 0x7f9b281067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:52914/0 (socket says 172.21.15.173:52914) 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.326+0000 7f9b26ffd700 1 -- 172.21.15.173:0/1173015761 learned_addr learned my addr 172.21.15.173:0/1173015761 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:26.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.326+0000 7f9b26ffd700 1 -- 172.21.15.173:0/1173015761 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c005040 con 0x7f9b280f6510 2024-02-19T22:04:26.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.327+0000 7f9b26ffd700 1 --2- 172.21.15.173:0/1173015761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b280f6510 0x7f9b281067e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f9b1c005010 tx=0x7f9b1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:26.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.327+0000 7f9b0f7fe700 1 -- 172.21.15.173:0/1173015761 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b1c009320 con 0x7f9b280f6510 2024-02-19T22:04:26.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.327+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1173015761 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b28106ed0 con 0x7f9b280f6510 2024-02-19T22:04:26.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.327+0000 7f9b0f7fe700 1 -- 172.21.15.173:0/1173015761 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b1c013380 con 0x7f9b280f6510 2024-02-19T22:04:26.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.327+0000 7f9b0f7fe700 1 -- 172.21.15.173:0/1173015761 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b1c014e50 con 0x7f9b280f6510 2024-02-19T22:04:26.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.327+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1173015761 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b28107360 con 0x7f9b280f6510 2024-02-19T22:04:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.328+0000 7f9b0f7fe700 1 -- 172.21.15.173:0/1173015761 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b1c0263a0 con 0x7f9b280f6510 2024-02-19T22:04:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.328+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1173015761 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b28045ad0 con 0x7f9b280f6510 2024-02-19T22:04:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.328+0000 7f9b0f7fe700 1 --2- 172.21.15.173:0/1173015761 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9b1002ff10 0x7f9b100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.328+0000 7f9b0f7fe700 1 -- 172.21.15.173:0/1173015761 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b1c041140 con 0x7f9b280f6510 2024-02-19T22:04:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.328+0000 7f9b267fc700 1 --2- 172.21.15.173:0/1173015761 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9b1002ff10 0x7f9b100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.329+0000 7f9b267fc700 1 --2- 172.21.15.173:0/1173015761 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9b1002ff10 0x7f9b100323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9b180096a0 tx=0x7f9b18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:26.332 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.333+0000 7f9b0f7fe700 1 -- 172.21.15.173:0/1173015761 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b1c01ccb0 con 0x7f9b280f6510 2024-02-19T22:04:26.548 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.548+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1173015761 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b28108210 con 0x7f9b280f6510 2024-02-19T22:04:26.548 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.549+0000 7f9b0f7fe700 1 -- 172.21.15.173:0/1173015761 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9b1c01cad0 con 0x7f9b280f6510 2024-02-19T22:04:26.549 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:26.549 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:26.550 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.551+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1173015761 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9b1002ff10 msgr2=0x7f9b100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:26.550 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.551+0000 7f9b2d4f6700 1 --2- 172.21.15.173:0/1173015761 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9b1002ff10 0x7f9b100323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9b180096a0 tx=0x7f9b18006b40).stop 2024-02-19T22:04:26.550 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.551+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1173015761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b280f6510 msgr2=0x7f9b281067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:26.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.551+0000 7f9b2d4f6700 1 --2- 172.21.15.173:0/1173015761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b280f6510 0x7f9b281067e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f9b1c005010 tx=0x7f9b1c00b040).stop 2024-02-19T22:04:26.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.551+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1173015761 shutdown_connections 2024-02-19T22:04:26.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.551+0000 7f9b2d4f6700 1 --2- 172.21.15.173:0/1173015761 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9b1002ff10 0x7f9b100323d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:26.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.551+0000 7f9b2d4f6700 1 --2- 172.21.15.173:0/1173015761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b280f6510 0x7f9b281067e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:26.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.551+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1173015761 >> 172.21.15.173:0/1173015761 conn(0x7f9b280f1c70 msgr2=0x7f9b280f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:26.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.551+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1173015761 shutdown_connections 2024-02-19T22:04:26.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:26.551+0000 7f9b2d4f6700 1 -- 172.21.15.173:0/1173015761 wait complete. 2024-02-19T22:04:26.551 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:26.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:24.528978+0000 mgr.smithi143.bissbc (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:27.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:27 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:26.550104+0000 mon.smithi143 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.173:0/1173015761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:28.486 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:28.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:26.529218+0000 mgr.smithi143.bissbc (mgr.14162) 312 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:28.515889+0000 mon.smithi143 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T22:04:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04cc015700 1 -- 172.21.15.173:0/930849326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04c40f5410 msgr2=0x7f04c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04cc015700 1 --2- 172.21.15.173:0/930849326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04c40f5410 0x7f04c40f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f04b4009fb0 tx=0x7f04b4009b30).stop 2024-02-19T22:04:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04cc015700 1 -- 172.21.15.173:0/930849326 shutdown_connections 2024-02-19T22:04:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04cc015700 1 --2- 172.21.15.173:0/930849326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04c40f5410 0x7f04c40f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04cc015700 1 -- 172.21.15.173:0/930849326 >> 172.21.15.173:0/930849326 conn(0x7f04c40f0b70 msgr2=0x7f04c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04cc015700 1 -- 172.21.15.173:0/930849326 shutdown_connections 2024-02-19T22:04:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04cc015700 1 -- 172.21.15.173:0/930849326 wait complete. 2024-02-19T22:04:30.593 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04cc015700 1 Processor -- start 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04cc015700 1 -- start start 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04cc015700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04c40f5410 0x7f04c41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04cc015700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4105bd0 con 0x7f04c40f5410 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.578+0000 7f04c9db1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04c40f5410 0x7f04c41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.579+0000 7f04c9db1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04c40f5410 0x7f04c41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:52920/0 (socket says 172.21.15.173:52920) 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.579+0000 7f04c9db1700 1 -- 172.21.15.173:0/2858743768 learned_addr learned my addr 172.21.15.173:0/2858743768 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.579+0000 7f04c9db1700 1 -- 172.21.15.173:0/2858743768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4005040 con 0x7f04c40f5410 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.579+0000 7f04c9db1700 1 --2- 172.21.15.173:0/2858743768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04c40f5410 0x7f04c41056e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f04b4005010 tx=0x7f04b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.579+0000 7f04ba7fc700 1 -- 172.21.15.173:0/2858743768 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04b4009320 con 0x7f04c40f5410 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.579+0000 7f04cc015700 1 -- 172.21.15.173:0/2858743768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c4105dd0 con 0x7f04c40f5410 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.579+0000 7f04ba7fc700 1 -- 172.21.15.173:0/2858743768 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04b4013380 con 0x7f04c40f5410 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.579+0000 7f04ba7fc700 1 -- 172.21.15.173:0/2858743768 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04b4014e50 con 0x7f04c40f5410 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.579+0000 7f04cc015700 1 -- 172.21.15.173:0/2858743768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c4106260 con 0x7f04c40f5410 2024-02-19T22:04:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.580+0000 7f04ba7fc700 1 -- 172.21.15.173:0/2858743768 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04b4014420 con 0x7f04c40f5410 2024-02-19T22:04:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.580+0000 7f04cc015700 1 -- 172.21.15.173:0/2858743768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c4045ad0 con 0x7f04c40f5410 2024-02-19T22:04:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.580+0000 7f04ba7fc700 1 --2- 172.21.15.173:0/2858743768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f04b002ff10 0x7f04b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.580+0000 7f04ba7fc700 1 -- 172.21.15.173:0/2858743768 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04b4040eb0 con 0x7f04c40f5410 2024-02-19T22:04:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.580+0000 7f04c95b0700 1 --2- 172.21.15.173:0/2858743768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f04b002ff10 0x7f04b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.581+0000 7f04c95b0700 1 --2- 172.21.15.173:0/2858743768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f04b002ff10 0x7f04b00323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f04c00096a0 tx=0x7f04c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:30.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.586+0000 7f04ba7fc700 1 -- 172.21.15.173:0/2858743768 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04b40095f0 con 0x7f04c40f5410 2024-02-19T22:04:30.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:28.529631+0000 mgr.smithi143.bissbc (mgr.14162) 313 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:30.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:29.328968+0000 mon.smithi143 (mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T22:04:30.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:29.329614+0000 mgr.smithi143.bissbc (mgr.14162) 314 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:30.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:29.334400+0000 mon.smithi143 (mon.0) 298 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' 2024-02-19T22:04:30.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cephadm 2024-02-19T22:04:29.336489+0000 mgr.smithi143.bissbc (mgr.14162) 315 : cephadm [INF] Deploying daemon alertmanager.smithi173 on smithi173 2024-02-19T22:04:30.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:29.514092+0000 mon.smithi143 (mon.0) 299 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-19T22:04:30.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.791+0000 7f04cc015700 1 -- 172.21.15.173:0/2858743768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04c4107110 con 0x7f04c40f5410 2024-02-19T22:04:30.792 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.793+0000 7f04ba7fc700 1 -- 172.21.15.173:0/2858743768 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f04b401db70 con 0x7f04c40f5410 2024-02-19T22:04:30.793 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:30.793 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:30.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.793+0000 7f04cc015700 1 -- 172.21.15.173:0/2858743768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f04b002ff10 msgr2=0x7f04b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:30.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.793+0000 7f04cc015700 1 --2- 172.21.15.173:0/2858743768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f04b002ff10 0x7f04b00323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f04c00096a0 tx=0x7f04c0006b40).stop 2024-02-19T22:04:30.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.793+0000 7f04cc015700 1 -- 172.21.15.173:0/2858743768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04c40f5410 msgr2=0x7f04c41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:30.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.793+0000 7f04cc015700 1 --2- 172.21.15.173:0/2858743768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04c40f5410 0x7f04c41056e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f04b4005010 tx=0x7f04b400b040).stop 2024-02-19T22:04:30.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.793+0000 7f04cc015700 1 -- 172.21.15.173:0/2858743768 shutdown_connections 2024-02-19T22:04:30.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.793+0000 7f04cc015700 1 --2- 172.21.15.173:0/2858743768 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f04b002ff10 0x7f04b00323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:30.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.793+0000 7f04cc015700 1 --2- 172.21.15.173:0/2858743768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04c40f5410 0x7f04c41056e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:30.794 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.793+0000 7f04cc015700 1 -- 172.21.15.173:0/2858743768 >> 172.21.15.173:0/2858743768 conn(0x7f04c40f0b70 msgr2=0x7f04c40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:30.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.794+0000 7f04cc015700 1 -- 172.21.15.173:0/2858743768 shutdown_connections 2024-02-19T22:04:30.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:30.794+0000 7f04cc015700 1 -- 172.21.15.173:0/2858743768 wait complete. 2024-02-19T22:04:30.795 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:31.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:31 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:30.794060+0000 mon.smithi143 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.173:0/2858743768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:32.549 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:32.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:32.659 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:31.329878+0000 mgr.smithi143.bissbc (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:34.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.778+0000 7f26b5957700 1 -- 172.21.15.173:0/449258485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b00f5410 msgr2=0x7f26b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:34.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.778+0000 7f26b5957700 1 --2- 172.21.15.173:0/449258485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b00f5410 0x7f26b00f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f26a0009fb0 tx=0x7f26a0009b30).stop 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.779+0000 7f26b5957700 1 -- 172.21.15.173:0/449258485 shutdown_connections 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.779+0000 7f26b5957700 1 --2- 172.21.15.173:0/449258485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b00f5410 0x7f26b00f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.779+0000 7f26b5957700 1 -- 172.21.15.173:0/449258485 >> 172.21.15.173:0/449258485 conn(0x7f26b00f0b70 msgr2=0x7f26b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.779+0000 7f26b5957700 1 -- 172.21.15.173:0/449258485 shutdown_connections 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.779+0000 7f26b5957700 1 -- 172.21.15.173:0/449258485 wait complete. 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.779+0000 7f26b5957700 1 Processor -- start 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.779+0000 7f26b5957700 1 -- start start 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.779+0000 7f26b5957700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b00f5410 0x7f26b0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.779+0000 7f26b5957700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b0105c10 con 0x7f26b00f5410 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.780+0000 7f26aeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b00f5410 0x7f26b0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.780+0000 7f26aeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b00f5410 0x7f26b0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:56108/0 (socket says 172.21.15.173:56108) 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.780+0000 7f26aeffd700 1 -- 172.21.15.173:0/3760725507 learned_addr learned my addr 172.21.15.173:0/3760725507 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:34.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.780+0000 7f26aeffd700 1 -- 172.21.15.173:0/3760725507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a0005040 con 0x7f26b00f5410 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.780+0000 7f26aeffd700 1 --2- 172.21.15.173:0/3760725507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b00f5410 0x7f26b0105720 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f26a0005010 tx=0x7f26a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.780+0000 7f26977fe700 1 -- 172.21.15.173:0/3760725507 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26a0009320 con 0x7f26b00f5410 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.780+0000 7f26b5957700 1 -- 172.21.15.173:0/3760725507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b0105e10 con 0x7f26b00f5410 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.780+0000 7f26977fe700 1 -- 172.21.15.173:0/3760725507 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26a0013380 con 0x7f26b00f5410 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.780+0000 7f26977fe700 1 -- 172.21.15.173:0/3760725507 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26a0014e50 con 0x7f26b00f5410 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.780+0000 7f26b5957700 1 -- 172.21.15.173:0/3760725507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b01062a0 con 0x7f26b00f5410 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.781+0000 7f26977fe700 1 -- 172.21.15.173:0/3760725507 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26a0014420 con 0x7f26b00f5410 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.781+0000 7f26b5957700 1 -- 172.21.15.173:0/3760725507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b0045ad0 con 0x7f26b00f5410 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.781+0000 7f26977fe700 1 --2- 172.21.15.173:0/3760725507 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f269802ff10 0x7f26980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.781+0000 7f26977fe700 1 -- 172.21.15.173:0/3760725507 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26a0040e30 con 0x7f26b00f5410 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.782+0000 7f26ae7fc700 1 --2- 172.21.15.173:0/3760725507 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f269802ff10 0x7f26980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.782+0000 7f26ae7fc700 1 --2- 172.21.15.173:0/3760725507 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f269802ff10 0x7f26980323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f26a40096a0 tx=0x7f26a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:34.790 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.787+0000 7f26977fe700 1 -- 172.21.15.173:0/3760725507 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26a00095f0 con 0x7f26b00f5410 2024-02-19T22:04:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:33.330205+0000 mgr.smithi143.bissbc (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:34.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.991+0000 7f26b5957700 1 -- 172.21.15.173:0/3760725507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26b0107150 con 0x7f26b00f5410 2024-02-19T22:04:34.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.992+0000 7f26977fe700 1 -- 172.21.15.173:0/3760725507 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f26a001db70 con 0x7f26b00f5410 2024-02-19T22:04:34.992 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:34.992 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:34.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.993+0000 7f26b5957700 1 -- 172.21.15.173:0/3760725507 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f269802ff10 msgr2=0x7f26980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:34.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.993+0000 7f26b5957700 1 --2- 172.21.15.173:0/3760725507 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f269802ff10 0x7f26980323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f26a40096a0 tx=0x7f26a4006b40).stop 2024-02-19T22:04:34.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.993+0000 7f26b5957700 1 -- 172.21.15.173:0/3760725507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b00f5410 msgr2=0x7f26b0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:34.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.993+0000 7f26b5957700 1 --2- 172.21.15.173:0/3760725507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b00f5410 0x7f26b0105720 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f26a0005010 tx=0x7f26a000b040).stop 2024-02-19T22:04:34.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.993+0000 7f26b5957700 1 -- 172.21.15.173:0/3760725507 shutdown_connections 2024-02-19T22:04:34.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.993+0000 7f26b5957700 1 --2- 172.21.15.173:0/3760725507 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f269802ff10 0x7f26980323d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:34.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.993+0000 7f26b5957700 1 --2- 172.21.15.173:0/3760725507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b00f5410 0x7f26b0105720 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:34.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.993+0000 7f26b5957700 1 -- 172.21.15.173:0/3760725507 >> 172.21.15.173:0/3760725507 conn(0x7f26b00f0b70 msgr2=0x7f26b00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:34.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.994+0000 7f26b5957700 1 -- 172.21.15.173:0/3760725507 shutdown_connections 2024-02-19T22:04:34.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:34.994+0000 7f26b5957700 1 -- 172.21.15.173:0/3760725507 wait complete. 2024-02-19T22:04:34.994 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:35.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:34.993064+0000 mon.smithi143 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.173:0/3760725507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:36.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:36.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:36.922 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:35.330525+0000 mgr.smithi143.bissbc (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:38.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.803+0000 7febb3f7e700 1 -- 172.21.15.173:0/3408625158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febac0f6510 msgr2=0x7febac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:38.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.803+0000 7febb3f7e700 1 --2- 172.21.15.173:0/3408625158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febac0f6510 0x7febac0f6930 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7feba0009fb0 tx=0x7feba0009b30).stop 2024-02-19T22:04:38.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.803+0000 7febb3f7e700 1 -- 172.21.15.173:0/3408625158 shutdown_connections 2024-02-19T22:04:38.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.803+0000 7febb3f7e700 1 --2- 172.21.15.173:0/3408625158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febac0f6510 0x7febac0f6930 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:38.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.803+0000 7febb3f7e700 1 -- 172.21.15.173:0/3408625158 >> 172.21.15.173:0/3408625158 conn(0x7febac0f1c70 msgr2=0x7febac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:38.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.803+0000 7febb3f7e700 1 -- 172.21.15.173:0/3408625158 shutdown_connections 2024-02-19T22:04:38.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.803+0000 7febb3f7e700 1 -- 172.21.15.173:0/3408625158 wait complete. 2024-02-19T22:04:38.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.804+0000 7febb3f7e700 1 Processor -- start 2024-02-19T22:04:38.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.804+0000 7febb3f7e700 1 -- start start 2024-02-19T22:04:38.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.804+0000 7febb3f7e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febac0f6510 0x7febac1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:38.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.804+0000 7febb3f7e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac106cd0 con 0x7febac0f6510 2024-02-19T22:04:38.819 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.804+0000 7febb1d1a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febac0f6510 0x7febac1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.804+0000 7febb1d1a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febac0f6510 0x7febac1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:56116/0 (socket says 172.21.15.173:56116) 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.804+0000 7febb1d1a700 1 -- 172.21.15.173:0/3715715276 learned_addr learned my addr 172.21.15.173:0/3715715276 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.804+0000 7febb1d1a700 1 -- 172.21.15.173:0/3715715276 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba0005040 con 0x7febac0f6510 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.804+0000 7febb1d1a700 1 --2- 172.21.15.173:0/3715715276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febac0f6510 0x7febac1067e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7feba0005010 tx=0x7feba000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.805+0000 7feb9e7fc700 1 -- 172.21.15.173:0/3715715276 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feba0009320 con 0x7febac0f6510 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.805+0000 7febb3f7e700 1 -- 172.21.15.173:0/3715715276 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febac106ed0 con 0x7febac0f6510 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.805+0000 7feb9e7fc700 1 -- 172.21.15.173:0/3715715276 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feba0013380 con 0x7febac0f6510 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.805+0000 7feb9e7fc700 1 -- 172.21.15.173:0/3715715276 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feba0014e50 con 0x7febac0f6510 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.805+0000 7febb3f7e700 1 -- 172.21.15.173:0/3715715276 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febac107360 con 0x7febac0f6510 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.805+0000 7feb9e7fc700 1 -- 172.21.15.173:0/3715715276 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feba00095f0 con 0x7febac0f6510 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.805+0000 7febb3f7e700 1 -- 172.21.15.173:0/3715715276 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febac045ad0 con 0x7febac0f6510 2024-02-19T22:04:38.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.806+0000 7feb9e7fc700 1 --2- 172.21.15.173:0/3715715276 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb9802ff10 0x7feb980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:38.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.806+0000 7feb9e7fc700 1 -- 172.21.15.173:0/3715715276 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feba0040e90 con 0x7febac0f6510 2024-02-19T22:04:38.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.806+0000 7febb1519700 1 --2- 172.21.15.173:0/3715715276 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb9802ff10 0x7feb980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:38.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.806+0000 7febb1519700 1 --2- 172.21.15.173:0/3715715276 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb9802ff10 0x7feb980323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7feba80096a0 tx=0x7feba8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:38.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:38.810+0000 7feb9e7fc700 1 -- 172.21.15.173:0/3715715276 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feba0014420 con 0x7febac0f6510 2024-02-19T22:04:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:37.330876+0000 mgr.smithi143.bissbc (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:39.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.017+0000 7febb3f7e700 1 -- 172.21.15.173:0/3715715276 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febac108210 con 0x7febac0f6510 2024-02-19T22:04:39.017 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.017+0000 7feb9e7fc700 1 -- 172.21.15.173:0/3715715276 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feba0010230 con 0x7febac0f6510 2024-02-19T22:04:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:39.017 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:39.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.019+0000 7febb3f7e700 1 -- 172.21.15.173:0/3715715276 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb9802ff10 msgr2=0x7feb980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:39.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.019+0000 7febb3f7e700 1 --2- 172.21.15.173:0/3715715276 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb9802ff10 0x7feb980323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7feba80096a0 tx=0x7feba8006b40).stop 2024-02-19T22:04:39.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.019+0000 7febb3f7e700 1 -- 172.21.15.173:0/3715715276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febac0f6510 msgr2=0x7febac1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:39.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.019+0000 7febb3f7e700 1 --2- 172.21.15.173:0/3715715276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febac0f6510 0x7febac1067e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7feba0005010 tx=0x7feba000b040).stop 2024-02-19T22:04:39.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.019+0000 7febb3f7e700 1 -- 172.21.15.173:0/3715715276 shutdown_connections 2024-02-19T22:04:39.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.019+0000 7febb3f7e700 1 --2- 172.21.15.173:0/3715715276 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7feb9802ff10 0x7feb980323d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:39.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.019+0000 7febb3f7e700 1 --2- 172.21.15.173:0/3715715276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febac0f6510 0x7febac1067e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:39.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.019+0000 7febb3f7e700 1 -- 172.21.15.173:0/3715715276 >> 172.21.15.173:0/3715715276 conn(0x7febac0f1c70 msgr2=0x7febac0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:39.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.019+0000 7febb3f7e700 1 -- 172.21.15.173:0/3715715276 shutdown_connections 2024-02-19T22:04:39.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:39.019+0000 7febb3f7e700 1 -- 172.21.15.173:0/3715715276 wait complete. 2024-02-19T22:04:39.019 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:39.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:39 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:39.018546+0000 mon.smithi143 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.173:0/3715715276' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:40.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:40.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:39.331235+0000 mgr.smithi143.bissbc (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:42.820 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.820+0000 7ff0af08f700 1 -- 172.21.15.173:0/821023942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0a80f6510 msgr2=0x7ff0a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:42.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.820+0000 7ff0af08f700 1 --2- 172.21.15.173:0/821023942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0a80f6510 0x7ff0a80f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7ff09c009fb0 tx=0x7ff09c009b30).stop 2024-02-19T22:04:42.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.820+0000 7ff0af08f700 1 -- 172.21.15.173:0/821023942 shutdown_connections 2024-02-19T22:04:42.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.820+0000 7ff0af08f700 1 --2- 172.21.15.173:0/821023942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0a80f6510 0x7ff0a80f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:42.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.820+0000 7ff0af08f700 1 -- 172.21.15.173:0/821023942 >> 172.21.15.173:0/821023942 conn(0x7ff0a80f1c70 msgr2=0x7ff0a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:42.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.820+0000 7ff0af08f700 1 -- 172.21.15.173:0/821023942 shutdown_connections 2024-02-19T22:04:42.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.820+0000 7ff0af08f700 1 -- 172.21.15.173:0/821023942 wait complete. 2024-02-19T22:04:42.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.820+0000 7ff0af08f700 1 Processor -- start 2024-02-19T22:04:42.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.821+0000 7ff0af08f700 1 -- start start 2024-02-19T22:04:42.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.821+0000 7ff0af08f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0a80f6510 0x7ff0a81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:42.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.821+0000 7ff0af08f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a8106cd0 con 0x7ff0a80f6510 2024-02-19T22:04:42.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.821+0000 7ff0ace2b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0a80f6510 0x7ff0a81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:42.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.821+0000 7ff0ace2b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0a80f6510 0x7ff0a81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:40656/0 (socket says 172.21.15.173:40656) 2024-02-19T22:04:42.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.821+0000 7ff0ace2b700 1 -- 172.21.15.173:0/1532306105 learned_addr learned my addr 172.21.15.173:0/1532306105 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:42.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.821+0000 7ff0ace2b700 1 -- 172.21.15.173:0/1532306105 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c005040 con 0x7ff0a80f6510 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.822+0000 7ff0ace2b700 1 --2- 172.21.15.173:0/1532306105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0a80f6510 0x7ff0a81067e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7ff09c005010 tx=0x7ff09c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.822+0000 7ff0a57fa700 1 -- 172.21.15.173:0/1532306105 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff09c009320 con 0x7ff0a80f6510 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.822+0000 7ff0af08f700 1 -- 172.21.15.173:0/1532306105 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a8106ed0 con 0x7ff0a80f6510 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.822+0000 7ff0a57fa700 1 -- 172.21.15.173:0/1532306105 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff09c013380 con 0x7ff0a80f6510 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.822+0000 7ff0a57fa700 1 -- 172.21.15.173:0/1532306105 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff09c014e50 con 0x7ff0a80f6510 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.822+0000 7ff0af08f700 1 -- 172.21.15.173:0/1532306105 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a8107360 con 0x7ff0a80f6510 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.823+0000 7ff0a57fa700 1 -- 172.21.15.173:0/1532306105 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff09c0095f0 con 0x7ff0a80f6510 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.823+0000 7ff0af08f700 1 -- 172.21.15.173:0/1532306105 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a8045ad0 con 0x7ff0a80f6510 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.823+0000 7ff0a57fa700 1 --2- 172.21.15.173:0/1532306105 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff09802ff10 0x7ff0980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.823+0000 7ff0a57fa700 1 -- 172.21.15.173:0/1532306105 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff09c040fa0 con 0x7ff0a80f6510 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.823+0000 7ff0a7fff700 1 --2- 172.21.15.173:0/1532306105 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff09802ff10 0x7ff0980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:42.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.823+0000 7ff0a7fff700 1 --2- 172.21.15.173:0/1532306105 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff09802ff10 0x7ff0980323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ff0940096a0 tx=0x7ff094006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:42.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:42.828+0000 7ff0a57fa700 1 -- 172.21.15.173:0/1532306105 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff09c014420 con 0x7ff0a80f6510 2024-02-19T22:04:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:41.331569+0000 mgr.smithi143.bissbc (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:43.031 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.032+0000 7ff0af08f700 1 -- 172.21.15.173:0/1532306105 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0a8108210 con 0x7ff0a80f6510 2024-02-19T22:04:43.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.032+0000 7ff0a57fa700 1 -- 172.21.15.173:0/1532306105 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff09c010230 con 0x7ff0a80f6510 2024-02-19T22:04:43.032 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:43.032 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:43.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.034+0000 7ff0af08f700 1 -- 172.21.15.173:0/1532306105 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff09802ff10 msgr2=0x7ff0980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:43.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.034+0000 7ff0af08f700 1 --2- 172.21.15.173:0/1532306105 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff09802ff10 0x7ff0980323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ff0940096a0 tx=0x7ff094006b40).stop 2024-02-19T22:04:43.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.034+0000 7ff0af08f700 1 -- 172.21.15.173:0/1532306105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0a80f6510 msgr2=0x7ff0a81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:43.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.034+0000 7ff0af08f700 1 --2- 172.21.15.173:0/1532306105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0a80f6510 0x7ff0a81067e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7ff09c005010 tx=0x7ff09c00b040).stop 2024-02-19T22:04:43.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.034+0000 7ff0af08f700 1 -- 172.21.15.173:0/1532306105 shutdown_connections 2024-02-19T22:04:43.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.034+0000 7ff0af08f700 1 --2- 172.21.15.173:0/1532306105 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff09802ff10 0x7ff0980323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:43.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.034+0000 7ff0af08f700 1 --2- 172.21.15.173:0/1532306105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0a80f6510 0x7ff0a81067e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:43.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.034+0000 7ff0af08f700 1 -- 172.21.15.173:0/1532306105 >> 172.21.15.173:0/1532306105 conn(0x7ff0a80f1c70 msgr2=0x7ff0a80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:43.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.034+0000 7ff0af08f700 1 -- 172.21.15.173:0/1532306105 shutdown_connections 2024-02-19T22:04:43.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:43.034+0000 7ff0af08f700 1 -- 172.21.15.173:0/1532306105 wait complete. 2024-02-19T22:04:43.035 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:43.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:43 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:43.033559+0000 mon.smithi143 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.173:0/1532306105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:43.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:43 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:43.331865+0000 mgr.smithi143.bissbc (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:44.667 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:44.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:46.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:45.332144+0000 mgr.smithi143.bissbc (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.729+0000 7f7595641700 1 -- 172.21.15.173:0/3297480134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7590060200 msgr2=0x7f75900fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.729+0000 7f7595641700 1 --2- 172.21.15.173:0/3297480134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7590060200 0x7f75900fb570 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f7578009fb0 tx=0x7f7578009b30).stop 2024-02-19T22:04:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.729+0000 7f7595641700 1 -- 172.21.15.173:0/3297480134 shutdown_connections 2024-02-19T22:04:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.729+0000 7f7595641700 1 --2- 172.21.15.173:0/3297480134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7590060200 0x7f75900fb570 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.729+0000 7f7595641700 1 -- 172.21.15.173:0/3297480134 >> 172.21.15.173:0/3297480134 conn(0x7f75900f0b70 msgr2=0x7f75900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.729+0000 7f7595641700 1 -- 172.21.15.173:0/3297480134 shutdown_connections 2024-02-19T22:04:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.729+0000 7f7595641700 1 -- 172.21.15.173:0/3297480134 wait complete. 2024-02-19T22:04:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.730+0000 7f7595641700 1 Processor -- start 2024-02-19T22:04:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.730+0000 7f7595641700 1 -- start start 2024-02-19T22:04:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.730+0000 7f7595641700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7590060200 0x7f75901089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.730+0000 7f7595641700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7590108e90 con 0x7f7590060200 2024-02-19T22:04:46.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.730+0000 7f758effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7590060200 0x7f75901089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:46.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.730+0000 7f758effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7590060200 0x7f75901089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:40688/0 (socket says 172.21.15.173:40688) 2024-02-19T22:04:46.731 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.730+0000 7f758effd700 1 -- 172.21.15.173:0/2132679299 learned_addr learned my addr 172.21.15.173:0/2132679299 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:46.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.731+0000 7f758effd700 1 -- 172.21.15.173:0/2132679299 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7578005040 con 0x7f7590060200 2024-02-19T22:04:46.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.731+0000 7f758effd700 1 --2- 172.21.15.173:0/2132679299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7590060200 0x7f75901089a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f75780089f0 tx=0x7f757800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:46.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.731+0000 7f75877fe700 1 -- 172.21.15.173:0/2132679299 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7578009320 con 0x7f7590060200 2024-02-19T22:04:46.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.731+0000 7f7595641700 1 -- 172.21.15.173:0/2132679299 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7590109090 con 0x7f7590060200 2024-02-19T22:04:46.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.731+0000 7f75877fe700 1 -- 172.21.15.173:0/2132679299 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7578013380 con 0x7f7590060200 2024-02-19T22:04:46.732 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.731+0000 7f75877fe700 1 -- 172.21.15.173:0/2132679299 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7578014e20 con 0x7f7590060200 2024-02-19T22:04:46.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.731+0000 7f7595641700 1 -- 172.21.15.173:0/2132679299 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7590109520 con 0x7f7590060200 2024-02-19T22:04:46.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.732+0000 7f75877fe700 1 -- 172.21.15.173:0/2132679299 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7578014420 con 0x7f7590060200 2024-02-19T22:04:46.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.732+0000 7f7595641700 1 -- 172.21.15.173:0/2132679299 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7590045ad0 con 0x7f7590060200 2024-02-19T22:04:46.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.732+0000 7f75877fe700 1 --2- 172.21.15.173:0/2132679299 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f757c02ff10 0x7f757c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:46.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.732+0000 7f75877fe700 1 -- 172.21.15.173:0/2132679299 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7578040eb0 con 0x7f7590060200 2024-02-19T22:04:46.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.732+0000 7f758e7fc700 1 --2- 172.21.15.173:0/2132679299 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f757c02ff10 0x7f757c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:46.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.733+0000 7f758e7fc700 1 --2- 172.21.15.173:0/2132679299 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f757c02ff10 0x7f757c0323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f75800096a0 tx=0x7f7580006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:46.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.737+0000 7f75877fe700 1 -- 172.21.15.173:0/2132679299 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7578014790 con 0x7f7590060200 2024-02-19T22:04:46.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.940+0000 7f7595641700 1 -- 172.21.15.173:0/2132679299 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f759010a370 con 0x7f7590060200 2024-02-19T22:04:46.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.941+0000 7f75877fe700 1 -- 172.21.15.173:0/2132679299 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f757801d7f0 con 0x7f7590060200 2024-02-19T22:04:46.940 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:46.941 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:46.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.942+0000 7f7595641700 1 -- 172.21.15.173:0/2132679299 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f757c02ff10 msgr2=0x7f757c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:46.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.942+0000 7f7595641700 1 --2- 172.21.15.173:0/2132679299 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f757c02ff10 0x7f757c0323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f75800096a0 tx=0x7f7580006b40).stop 2024-02-19T22:04:46.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.942+0000 7f7595641700 1 -- 172.21.15.173:0/2132679299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7590060200 msgr2=0x7f75901089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:46.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.942+0000 7f7595641700 1 --2- 172.21.15.173:0/2132679299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7590060200 0x7f75901089a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f75780089f0 tx=0x7f757800b040).stop 2024-02-19T22:04:46.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.943+0000 7f7595641700 1 -- 172.21.15.173:0/2132679299 shutdown_connections 2024-02-19T22:04:46.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.943+0000 7f7595641700 1 --2- 172.21.15.173:0/2132679299 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f757c02ff10 0x7f757c0323d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:46.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.943+0000 7f7595641700 1 --2- 172.21.15.173:0/2132679299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7590060200 0x7f75901089a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:46.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.943+0000 7f7595641700 1 -- 172.21.15.173:0/2132679299 >> 172.21.15.173:0/2132679299 conn(0x7f75900f0b70 msgr2=0x7f75900f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:46.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.943+0000 7f7595641700 1 -- 172.21.15.173:0/2132679299 shutdown_connections 2024-02-19T22:04:46.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:46.943+0000 7f7595641700 1 -- 172.21.15.173:0/2132679299 wait complete. 2024-02-19T22:04:46.943 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:47.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:47 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:46.942005+0000 mon.smithi143 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.173:0/2132679299' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:48.787 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:48.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:47.332412+0000 mgr.smithi143.bissbc (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:50.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.738+0000 7f007ffff700 1 -- 172.21.15.173:0/790634000 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00800f51a0 msgr2=0x7f00800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:50.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.738+0000 7f007ffff700 1 --2- 172.21.15.173:0/790634000 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00800f51a0 0x7f00800f55c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f0068009fb0 tx=0x7f0068009b30).stop 2024-02-19T22:04:50.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.738+0000 7f007ffff700 1 -- 172.21.15.173:0/790634000 shutdown_connections 2024-02-19T22:04:50.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.738+0000 7f007ffff700 1 --2- 172.21.15.173:0/790634000 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00800f51a0 0x7f00800f55c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:50.739 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.738+0000 7f007ffff700 1 -- 172.21.15.173:0/790634000 >> 172.21.15.173:0/790634000 conn(0x7f00800f09a0 msgr2=0x7f00800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.738+0000 7f007ffff700 1 -- 172.21.15.173:0/790634000 shutdown_connections 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.738+0000 7f007ffff700 1 -- 172.21.15.173:0/790634000 wait complete. 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.739+0000 7f007ffff700 1 Processor -- start 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.739+0000 7f007ffff700 1 -- start start 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.739+0000 7f007ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00800f51a0 0x7f0080109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.739+0000 7f007ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0080060280 con 0x7f00800f51a0 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.739+0000 7f007effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00800f51a0 0x7f0080109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.739+0000 7f007effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00800f51a0 0x7f0080109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:40708/0 (socket says 172.21.15.173:40708) 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.739+0000 7f007effd700 1 -- 172.21.15.173:0/1567391172 learned_addr learned my addr 172.21.15.173:0/1567391172 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.739+0000 7f007effd700 1 -- 172.21.15.173:0/1567391172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0068005040 con 0x7f00800f51a0 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.739+0000 7f007effd700 1 --2- 172.21.15.173:0/1567391172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00800f51a0 0x7f0080109290 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f0068009830 tx=0x7f006800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.740+0000 7f00777fe700 1 -- 172.21.15.173:0/1567391172 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0068009320 con 0x7f00800f51a0 2024-02-19T22:04:50.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.740+0000 7f007ffff700 1 -- 172.21.15.173:0/1567391172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0080060480 con 0x7f00800f51a0 2024-02-19T22:04:50.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.740+0000 7f00777fe700 1 -- 172.21.15.173:0/1567391172 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0068013380 con 0x7f00800f51a0 2024-02-19T22:04:50.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.740+0000 7f00777fe700 1 -- 172.21.15.173:0/1567391172 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0068014420 con 0x7f00800f51a0 2024-02-19T22:04:50.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.740+0000 7f007ffff700 1 -- 172.21.15.173:0/1567391172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00801099c0 con 0x7f00800f51a0 2024-02-19T22:04:50.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.740+0000 7f00777fe700 1 -- 172.21.15.173:0/1567391172 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f006801e420 con 0x7f00800f51a0 2024-02-19T22:04:50.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.740+0000 7f007ffff700 1 -- 172.21.15.173:0/1567391172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0080045ad0 con 0x7f00800f51a0 2024-02-19T22:04:50.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.741+0000 7f00777fe700 1 --2- 172.21.15.173:0/1567391172 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f006c02ffb0 0x7f006c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:50.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.741+0000 7f00777fe700 1 -- 172.21.15.173:0/1567391172 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0068040df0 con 0x7f00800f51a0 2024-02-19T22:04:50.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.741+0000 7f007e7fc700 1 --2- 172.21.15.173:0/1567391172 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f006c02ffb0 0x7f006c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:50.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.741+0000 7f007e7fc700 1 --2- 172.21.15.173:0/1567391172 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f006c02ffb0 0x7f006c032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f00700096a0 tx=0x7f0070006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:50.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.745+0000 7f00777fe700 1 -- 172.21.15.173:0/1567391172 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f006801c970 con 0x7f00800f51a0 2024-02-19T22:04:50.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:50 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:49.332724+0000 mgr.smithi143.bissbc (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:50.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.949+0000 7f007ffff700 1 -- 172.21.15.173:0/1567391172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00800fad30 con 0x7f00800f51a0 2024-02-19T22:04:50.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.949+0000 7f00777fe700 1 -- 172.21.15.173:0/1567391172 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f006802c2d0 con 0x7f00800f51a0 2024-02-19T22:04:50.949 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:50.949 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:50.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.951+0000 7f007ffff700 1 -- 172.21.15.173:0/1567391172 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f006c02ffb0 msgr2=0x7f006c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:50.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.951+0000 7f007ffff700 1 --2- 172.21.15.173:0/1567391172 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f006c02ffb0 0x7f006c032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f00700096a0 tx=0x7f0070006b40).stop 2024-02-19T22:04:50.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.951+0000 7f007ffff700 1 -- 172.21.15.173:0/1567391172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00800f51a0 msgr2=0x7f0080109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:50.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.951+0000 7f007ffff700 1 --2- 172.21.15.173:0/1567391172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00800f51a0 0x7f0080109290 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f0068009830 tx=0x7f006800b040).stop 2024-02-19T22:04:50.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.951+0000 7f007ffff700 1 -- 172.21.15.173:0/1567391172 shutdown_connections 2024-02-19T22:04:50.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.951+0000 7f007ffff700 1 --2- 172.21.15.173:0/1567391172 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f006c02ffb0 0x7f006c032470 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:50.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.951+0000 7f007ffff700 1 --2- 172.21.15.173:0/1567391172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00800f51a0 0x7f0080109290 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:50.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.951+0000 7f007ffff700 1 -- 172.21.15.173:0/1567391172 >> 172.21.15.173:0/1567391172 conn(0x7f00800f09a0 msgr2=0x7f00800f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:50.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.951+0000 7f007ffff700 1 -- 172.21.15.173:0/1567391172 shutdown_connections 2024-02-19T22:04:50.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:50.951+0000 7f007ffff700 1 -- 172.21.15.173:0/1567391172 wait complete. 2024-02-19T22:04:50.952 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:51.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:51 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:50.950790+0000 mon.smithi143 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.173:0/1567391172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:52.568 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:52.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:52.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:51.333123+0000 mgr.smithi143.bissbc (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:54.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.624+0000 7f6857172700 1 -- 172.21.15.173:0/482451417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68500f6510 msgr2=0x7f68500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.624+0000 7f6857172700 1 --2- 172.21.15.173:0/482451417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68500f6510 0x7f68500f6930 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f6844009fb0 tx=0x7f6844009b30).stop 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.624+0000 7f6857172700 1 -- 172.21.15.173:0/482451417 shutdown_connections 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.624+0000 7f6857172700 1 --2- 172.21.15.173:0/482451417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68500f6510 0x7f68500f6930 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.624+0000 7f6857172700 1 -- 172.21.15.173:0/482451417 >> 172.21.15.173:0/482451417 conn(0x7f68500f1c70 msgr2=0x7f68500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.624+0000 7f6857172700 1 -- 172.21.15.173:0/482451417 shutdown_connections 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.624+0000 7f6857172700 1 -- 172.21.15.173:0/482451417 wait complete. 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.625+0000 7f6857172700 1 Processor -- start 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.625+0000 7f6857172700 1 -- start start 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.625+0000 7f6857172700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68500f6510 0x7f68501067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.625+0000 7f6857172700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6850106cd0 con 0x7f68500f6510 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.625+0000 7f6854f0e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68500f6510 0x7f68501067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.625+0000 7f6854f0e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68500f6510 0x7f68501067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:45058/0 (socket says 172.21.15.173:45058) 2024-02-19T22:04:54.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.625+0000 7f6854f0e700 1 -- 172.21.15.173:0/1829878117 learned_addr learned my addr 172.21.15.173:0/1829878117 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:54.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.625+0000 7f6854f0e700 1 -- 172.21.15.173:0/1829878117 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6844005040 con 0x7f68500f6510 2024-02-19T22:04:54.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.626+0000 7f6854f0e700 1 --2- 172.21.15.173:0/1829878117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68500f6510 0x7f68501067e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f6844005010 tx=0x7f684400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:54.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.626+0000 7f684d7fa700 1 -- 172.21.15.173:0/1829878117 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6844009320 con 0x7f68500f6510 2024-02-19T22:04:54.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.626+0000 7f684d7fa700 1 -- 172.21.15.173:0/1829878117 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6844013380 con 0x7f68500f6510 2024-02-19T22:04:54.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.626+0000 7f6857172700 1 -- 172.21.15.173:0/1829878117 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6850106ed0 con 0x7f68500f6510 2024-02-19T22:04:54.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.626+0000 7f684d7fa700 1 -- 172.21.15.173:0/1829878117 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6844014e50 con 0x7f68500f6510 2024-02-19T22:04:54.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.626+0000 7f6857172700 1 -- 172.21.15.173:0/1829878117 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6850107360 con 0x7f68500f6510 2024-02-19T22:04:54.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.626+0000 7f684d7fa700 1 -- 172.21.15.173:0/1829878117 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68440095f0 con 0x7f68500f6510 2024-02-19T22:04:54.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.627+0000 7f684d7fa700 1 --2- 172.21.15.173:0/1829878117 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f684002ff10 0x7f68400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:54.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.627+0000 7f684d7fa700 1 -- 172.21.15.173:0/1829878117 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6844040e90 con 0x7f68500f6510 2024-02-19T22:04:54.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.627+0000 7f6857172700 1 -- 172.21.15.173:0/1829878117 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6850045ad0 con 0x7f68500f6510 2024-02-19T22:04:54.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.627+0000 7f684ffff700 1 --2- 172.21.15.173:0/1829878117 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f684002ff10 0x7f68400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:54.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.627+0000 7f684ffff700 1 --2- 172.21.15.173:0/1829878117 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f684002ff10 0x7f68400323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f683c0096a0 tx=0x7f683c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:54.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.632+0000 7f684d7fa700 1 -- 172.21.15.173:0/1829878117 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6844014420 con 0x7f68500f6510 2024-02-19T22:04:54.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.841+0000 7f6857172700 1 -- 172.21.15.173:0/1829878117 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6850108080 con 0x7f68500f6510 2024-02-19T22:04:54.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.841+0000 7f684d7fa700 1 -- 172.21.15.173:0/1829878117 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6844010230 con 0x7f68500f6510 2024-02-19T22:04:54.843 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:54.843 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:54.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.843+0000 7f6857172700 1 -- 172.21.15.173:0/1829878117 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f684002ff10 msgr2=0x7f68400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:54.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.843+0000 7f6857172700 1 --2- 172.21.15.173:0/1829878117 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f684002ff10 0x7f68400323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f683c0096a0 tx=0x7f683c006b40).stop 2024-02-19T22:04:54.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.843+0000 7f6857172700 1 -- 172.21.15.173:0/1829878117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68500f6510 msgr2=0x7f68501067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:54.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.843+0000 7f6857172700 1 --2- 172.21.15.173:0/1829878117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68500f6510 0x7f68501067e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f6844005010 tx=0x7f684400b040).stop 2024-02-19T22:04:54.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.843+0000 7f6857172700 1 -- 172.21.15.173:0/1829878117 shutdown_connections 2024-02-19T22:04:54.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.843+0000 7f6857172700 1 --2- 172.21.15.173:0/1829878117 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f684002ff10 0x7f68400323d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:54.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.843+0000 7f6857172700 1 --2- 172.21.15.173:0/1829878117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68500f6510 0x7f68501067e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:54.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.843+0000 7f6857172700 1 -- 172.21.15.173:0/1829878117 >> 172.21.15.173:0/1829878117 conn(0x7f68500f1c70 msgr2=0x7f68500f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:54.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.843+0000 7f6857172700 1 -- 172.21.15.173:0/1829878117 shutdown_connections 2024-02-19T22:04:54.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:54.843+0000 7f6857172700 1 -- 172.21.15.173:0/1829878117 wait complete. 2024-02-19T22:04:54.846 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:53.333470+0000 mgr.smithi143.bissbc (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:55.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:55 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:54.842613+0000 mon.smithi143 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.173:0/1829878117' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:56.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:56.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:55.333807+0000 mgr.smithi143.bissbc (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:58.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.734+0000 7f8a735ae700 1 -- 172.21.15.173:0/4141644954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6c0f5410 msgr2=0x7f8a6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:58.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.734+0000 7f8a735ae700 1 --2- 172.21.15.173:0/4141644954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c0f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f8a68009fb0 tx=0x7f8a68009b30).stop 2024-02-19T22:04:58.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.734+0000 7f8a735ae700 1 -- 172.21.15.173:0/4141644954 shutdown_connections 2024-02-19T22:04:58.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.734+0000 7f8a735ae700 1 --2- 172.21.15.173:0/4141644954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c0f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:58.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.734+0000 7f8a735ae700 1 -- 172.21.15.173:0/4141644954 >> 172.21.15.173:0/4141644954 conn(0x7f8a6c0f0b70 msgr2=0x7f8a6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:58.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.734+0000 7f8a735ae700 1 -- 172.21.15.173:0/4141644954 shutdown_connections 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.734+0000 7f8a735ae700 1 -- 172.21.15.173:0/4141644954 wait complete. 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.735+0000 7f8a735ae700 1 Processor -- start 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.735+0000 7f8a735ae700 1 -- start start 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.735+0000 7f8a735ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.735+0000 7f8a735ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c0fb960 con 0x7f8a6c0f5410 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.735+0000 7f8a7134a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.735+0000 7f8a7134a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:45082/0 (socket says 172.21.15.173:45082) 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.735+0000 7f8a7134a700 1 -- 172.21.15.173:0/1326688781 learned_addr learned my addr 172.21.15.173:0/1326688781 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.735+0000 7f8a7134a700 1 -- 172.21.15.173:0/1326688781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a68005040 con 0x7f8a6c0f5410 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.735+0000 7f8a7134a700 1 --2- 172.21.15.173:0/1326688781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c0fd4e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f8a680089f0 tx=0x7f8a6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.736+0000 7f8a61ffb700 1 -- 172.21.15.173:0/1326688781 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a68009320 con 0x7f8a6c0f5410 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.736+0000 7f8a735ae700 1 -- 172.21.15.173:0/1326688781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6c0fbb00 con 0x7f8a6c0f5410 2024-02-19T22:04:58.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.736+0000 7f8a61ffb700 1 -- 172.21.15.173:0/1326688781 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a68013380 con 0x7f8a6c0f5410 2024-02-19T22:04:58.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.736+0000 7f8a61ffb700 1 -- 172.21.15.173:0/1326688781 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a6801ed80 con 0x7f8a6c0f5410 2024-02-19T22:04:58.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.736+0000 7f8a735ae700 1 -- 172.21.15.173:0/1326688781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c0fbf90 con 0x7f8a6c0f5410 2024-02-19T22:04:58.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.737+0000 7f8a61ffb700 1 -- 172.21.15.173:0/1326688781 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a68014420 con 0x7f8a6c0f5410 2024-02-19T22:04:58.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.737+0000 7f8a61ffb700 1 --2- 172.21.15.173:0/1326688781 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a5802ff60 0x7f8a58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:58.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.737+0000 7f8a61ffb700 1 -- 172.21.15.173:0/1326688781 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a68040e30 con 0x7f8a6c0f5410 2024-02-19T22:04:58.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.737+0000 7f8a735ae700 1 -- 172.21.15.173:0/1326688781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6c059100 con 0x7f8a6c0f5410 2024-02-19T22:04:58.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.737+0000 7f8a70b49700 1 --2- 172.21.15.173:0/1326688781 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a5802ff60 0x7f8a58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:58.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.737+0000 7f8a70b49700 1 --2- 172.21.15.173:0/1326688781 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a5802ff60 0x7f8a58032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8a5c0096a0 tx=0x7f8a5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:58.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.742+0000 7f8a61ffb700 1 -- 172.21.15.173:0/1326688781 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a6801c910 con 0x7f8a6c0f5410 2024-02-19T22:04:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:57.334149+0000 mgr.smithi143.bissbc (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:58.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.949+0000 7f8a735ae700 1 -- 172.21.15.173:0/1326688781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a6c0fc690 con 0x7f8a6c0f5410 2024-02-19T22:04:58.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.950+0000 7f8a61ffb700 1 -- 172.21.15.173:0/1326688781 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8a6801c320 con 0x7f8a6c0f5410 2024-02-19T22:04:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:04:58.950 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:58.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.952+0000 7f8a735ae700 1 -- 172.21.15.173:0/1326688781 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a5802ff60 msgr2=0x7f8a58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:58.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.952+0000 7f8a735ae700 1 --2- 172.21.15.173:0/1326688781 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a5802ff60 0x7f8a58032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8a5c0096a0 tx=0x7f8a5c006b40).stop 2024-02-19T22:04:58.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.952+0000 7f8a735ae700 1 -- 172.21.15.173:0/1326688781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6c0f5410 msgr2=0x7f8a6c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.952+0000 7f8a735ae700 1 --2- 172.21.15.173:0/1326688781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c0fd4e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f8a680089f0 tx=0x7f8a6800b040).stop 2024-02-19T22:04:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.952+0000 7f8a735ae700 1 -- 172.21.15.173:0/1326688781 shutdown_connections 2024-02-19T22:04:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.952+0000 7f8a735ae700 1 --2- 172.21.15.173:0/1326688781 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a5802ff60 0x7f8a58032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.952+0000 7f8a735ae700 1 --2- 172.21.15.173:0/1326688781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c0fd4e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.952+0000 7f8a735ae700 1 -- 172.21.15.173:0/1326688781 >> 172.21.15.173:0/1326688781 conn(0x7f8a6c0f0b70 msgr2=0x7f8a6c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.952+0000 7f8a735ae700 1 -- 172.21.15.173:0/1326688781 shutdown_connections 2024-02-19T22:04:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:04:58.952+0000 7f8a735ae700 1 -- 172.21.15.173:0/1326688781 wait complete. 2024-02-19T22:04:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:04:59.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:04:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:04:58.951411+0000 mon.smithi143 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.173:0/1326688781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:00.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:00.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:04:59.334523+0000 mgr.smithi143.bissbc (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:02.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:02 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:01.334864+0000 mgr.smithi143.bissbc (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:02.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.910+0000 7fb5a5872700 1 -- 172.21.15.173:0/2466473551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5a00f6510 msgr2=0x7fb5a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:02.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.910+0000 7fb5a5872700 1 --2- 172.21.15.173:0/2466473551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5a00f6510 0x7fb5a00f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fb594009fb0 tx=0x7fb594009b30).stop 2024-02-19T22:05:02.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.910+0000 7fb5a5872700 1 -- 172.21.15.173:0/2466473551 shutdown_connections 2024-02-19T22:05:02.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.910+0000 7fb5a5872700 1 --2- 172.21.15.173:0/2466473551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5a00f6510 0x7fb5a00f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:02.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.910+0000 7fb5a5872700 1 -- 172.21.15.173:0/2466473551 >> 172.21.15.173:0/2466473551 conn(0x7fb5a00f1c70 msgr2=0x7fb5a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:02.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.910+0000 7fb5a5872700 1 -- 172.21.15.173:0/2466473551 shutdown_connections 2024-02-19T22:05:02.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.910+0000 7fb5a5872700 1 -- 172.21.15.173:0/2466473551 wait complete. 2024-02-19T22:05:02.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.911+0000 7fb5a5872700 1 Processor -- start 2024-02-19T22:05:02.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.911+0000 7fb5a5872700 1 -- start start 2024-02-19T22:05:02.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.911+0000 7fb5a5872700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5a00f6510 0x7fb5a01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:02.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.911+0000 7fb5a5872700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a0106cd0 con 0x7fb5a00f6510 2024-02-19T22:05:02.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.911+0000 7fb59effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5a00f6510 0x7fb5a01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:02.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.911+0000 7fb59effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5a00f6510 0x7fb5a01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41380/0 (socket says 172.21.15.173:41380) 2024-02-19T22:05:02.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.911+0000 7fb59effd700 1 -- 172.21.15.173:0/3983158881 learned_addr learned my addr 172.21.15.173:0/3983158881 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:02.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.911+0000 7fb59effd700 1 -- 172.21.15.173:0/3983158881 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb594005040 con 0x7fb5a00f6510 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.911+0000 7fb59effd700 1 --2- 172.21.15.173:0/3983158881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5a00f6510 0x7fb5a01067e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb594005010 tx=0x7fb59400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.912+0000 7fb5877fe700 1 -- 172.21.15.173:0/3983158881 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb594009320 con 0x7fb5a00f6510 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.912+0000 7fb5a5872700 1 -- 172.21.15.173:0/3983158881 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a0106ed0 con 0x7fb5a00f6510 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.912+0000 7fb5877fe700 1 -- 172.21.15.173:0/3983158881 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb594013380 con 0x7fb5a00f6510 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.912+0000 7fb5877fe700 1 -- 172.21.15.173:0/3983158881 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb594014e50 con 0x7fb5a00f6510 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.912+0000 7fb5a5872700 1 -- 172.21.15.173:0/3983158881 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a0107360 con 0x7fb5a00f6510 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.912+0000 7fb5877fe700 1 -- 172.21.15.173:0/3983158881 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5940095f0 con 0x7fb5a00f6510 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.913+0000 7fb5a5872700 1 -- 172.21.15.173:0/3983158881 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a0045ad0 con 0x7fb5a00f6510 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.913+0000 7fb5877fe700 1 --2- 172.21.15.173:0/3983158881 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb58802ff10 0x7fb5880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.913+0000 7fb5877fe700 1 -- 172.21.15.173:0/3983158881 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb594040e90 con 0x7fb5a00f6510 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.913+0000 7fb59e7fc700 1 --2- 172.21.15.173:0/3983158881 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb58802ff10 0x7fb5880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.913+0000 7fb59e7fc700 1 --2- 172.21.15.173:0/3983158881 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb58802ff10 0x7fb5880323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb5900096a0 tx=0x7fb590006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:02.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:02.918+0000 7fb5877fe700 1 -- 172.21.15.173:0/3983158881 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb594014420 con 0x7fb5a00f6510 2024-02-19T22:05:03.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.124+0000 7fb5a5872700 1 -- 172.21.15.173:0/3983158881 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5a0108210 con 0x7fb5a00f6510 2024-02-19T22:05:03.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.125+0000 7fb5877fe700 1 -- 172.21.15.173:0/3983158881 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb594010230 con 0x7fb5a00f6510 2024-02-19T22:05:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:03.125 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:03.126 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.126+0000 7fb5a5872700 1 -- 172.21.15.173:0/3983158881 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb58802ff10 msgr2=0x7fb5880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:03.126 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.126+0000 7fb5a5872700 1 --2- 172.21.15.173:0/3983158881 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb58802ff10 0x7fb5880323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb5900096a0 tx=0x7fb590006b40).stop 2024-02-19T22:05:03.126 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.126+0000 7fb5a5872700 1 -- 172.21.15.173:0/3983158881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5a00f6510 msgr2=0x7fb5a01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:03.126 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.126+0000 7fb5a5872700 1 --2- 172.21.15.173:0/3983158881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5a00f6510 0x7fb5a01067e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb594005010 tx=0x7fb59400b040).stop 2024-02-19T22:05:03.126 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.126+0000 7fb5a5872700 1 -- 172.21.15.173:0/3983158881 shutdown_connections 2024-02-19T22:05:03.126 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.126+0000 7fb5a5872700 1 --2- 172.21.15.173:0/3983158881 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb58802ff10 0x7fb5880323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:03.126 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.127+0000 7fb5a5872700 1 --2- 172.21.15.173:0/3983158881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5a00f6510 0x7fb5a01067e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:03.127 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.127+0000 7fb5a5872700 1 -- 172.21.15.173:0/3983158881 >> 172.21.15.173:0/3983158881 conn(0x7fb5a00f1c70 msgr2=0x7fb5a00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:03.127 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.127+0000 7fb5a5872700 1 -- 172.21.15.173:0/3983158881 shutdown_connections 2024-02-19T22:05:03.127 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:03.127+0000 7fb5a5872700 1 -- 172.21.15.173:0/3983158881 wait complete. 2024-02-19T22:05:03.127 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:03.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:03 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:03.126044+0000 mon.smithi143 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.173:0/3983158881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:04.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:04.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:03.335196+0000 mgr.smithi143.bissbc (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:06.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.845+0000 7f474ca76700 1 -- 172.21.15.173:0/2518141152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47480f5410 msgr2=0x7f47480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:06.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.845+0000 7f474ca76700 1 --2- 172.21.15.173:0/2518141152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47480f5410 0x7f47480f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f4730009fb0 tx=0x7f4730009b30).stop 2024-02-19T22:05:06.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474ca76700 1 -- 172.21.15.173:0/2518141152 shutdown_connections 2024-02-19T22:05:06.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474ca76700 1 --2- 172.21.15.173:0/2518141152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47480f5410 0x7f47480f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:06.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474ca76700 1 -- 172.21.15.173:0/2518141152 >> 172.21.15.173:0/2518141152 conn(0x7f47480f0b70 msgr2=0x7f47480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:06.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474ca76700 1 -- 172.21.15.173:0/2518141152 shutdown_connections 2024-02-19T22:05:06.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474ca76700 1 -- 172.21.15.173:0/2518141152 wait complete. 2024-02-19T22:05:06.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474ca76700 1 Processor -- start 2024-02-19T22:05:06.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474ca76700 1 -- start start 2024-02-19T22:05:06.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474ca76700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47480f5410 0x7f4748105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:06.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474ca76700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4748105c10 con 0x7f47480f5410 2024-02-19T22:05:06.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47480f5410 0x7f4748105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:06.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47480f5410 0x7f4748105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41396/0 (socket says 172.21.15.173:41396) 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.846+0000 7f474659c700 1 -- 172.21.15.173:0/4103965022 learned_addr learned my addr 172.21.15.173:0/4103965022 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.847+0000 7f474659c700 1 -- 172.21.15.173:0/4103965022 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4730005040 con 0x7f47480f5410 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.847+0000 7f474659c700 1 --2- 172.21.15.173:0/4103965022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47480f5410 0x7f4748105720 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4730005010 tx=0x7f473000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.847+0000 7f473effd700 1 -- 172.21.15.173:0/4103965022 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4730009320 con 0x7f47480f5410 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.847+0000 7f473effd700 1 -- 172.21.15.173:0/4103965022 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4730013380 con 0x7f47480f5410 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.847+0000 7f474ca76700 1 -- 172.21.15.173:0/4103965022 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4748105e10 con 0x7f47480f5410 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.847+0000 7f473effd700 1 -- 172.21.15.173:0/4103965022 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4730014e50 con 0x7f47480f5410 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.847+0000 7f474ca76700 1 -- 172.21.15.173:0/4103965022 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47481062a0 con 0x7f47480f5410 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.848+0000 7f473effd700 1 -- 172.21.15.173:0/4103965022 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47300263a0 con 0x7f47480f5410 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.848+0000 7f473effd700 1 --2- 172.21.15.173:0/4103965022 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f473402ff10 0x7f47340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.848+0000 7f473effd700 1 -- 172.21.15.173:0/4103965022 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4730040e60 con 0x7f47480f5410 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.848+0000 7f474ca76700 1 -- 172.21.15.173:0/4103965022 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4748045ad0 con 0x7f47480f5410 2024-02-19T22:05:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.848+0000 7f4745d9b700 1 --2- 172.21.15.173:0/4103965022 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f473402ff10 0x7f47340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:06.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.849+0000 7f4745d9b700 1 --2- 172.21.15.173:0/4103965022 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f473402ff10 0x7f47340323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f47380096a0 tx=0x7f4738006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:06.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:06.853+0000 7f473effd700 1 -- 172.21.15.173:0/4103965022 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4730012920 con 0x7f47480f5410 2024-02-19T22:05:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:05.335589+0000 mgr.smithi143.bissbc (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:07.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.060+0000 7f474ca76700 1 -- 172.21.15.173:0/4103965022 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4748106ff0 con 0x7f47480f5410 2024-02-19T22:05:07.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.060+0000 7f473effd700 1 -- 172.21.15.173:0/4103965022 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f473000f340 con 0x7f47480f5410 2024-02-19T22:05:07.060 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:07.060 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.062+0000 7f474ca76700 1 -- 172.21.15.173:0/4103965022 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f473402ff10 msgr2=0x7f47340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.062+0000 7f474ca76700 1 --2- 172.21.15.173:0/4103965022 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f473402ff10 0x7f47340323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f47380096a0 tx=0x7f4738006b40).stop 2024-02-19T22:05:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.062+0000 7f474ca76700 1 -- 172.21.15.173:0/4103965022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47480f5410 msgr2=0x7f4748105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.062+0000 7f474ca76700 1 --2- 172.21.15.173:0/4103965022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47480f5410 0x7f4748105720 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4730005010 tx=0x7f473000b040).stop 2024-02-19T22:05:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.062+0000 7f474ca76700 1 -- 172.21.15.173:0/4103965022 shutdown_connections 2024-02-19T22:05:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.062+0000 7f474ca76700 1 --2- 172.21.15.173:0/4103965022 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f473402ff10 0x7f47340323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.062+0000 7f474ca76700 1 --2- 172.21.15.173:0/4103965022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47480f5410 0x7f4748105720 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.062+0000 7f474ca76700 1 -- 172.21.15.173:0/4103965022 >> 172.21.15.173:0/4103965022 conn(0x7f47480f0b70 msgr2=0x7f47480f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.062+0000 7f474ca76700 1 -- 172.21.15.173:0/4103965022 shutdown_connections 2024-02-19T22:05:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:07.062+0000 7f474ca76700 1 -- 172.21.15.173:0/4103965022 wait complete. 2024-02-19T22:05:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:07.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:07 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:07.061429+0000 mon.smithi143 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.173:0/4103965022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:08.677 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:08.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:07.336001+0000 mgr.smithi143.bissbc (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:09.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:09 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:09.336274+0000 mgr.smithi143.bissbc (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.588+0000 7f934304d700 1 -- 172.21.15.173:0/663636342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0f5410 msgr2=0x7f933c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.588+0000 7f934304d700 1 --2- 172.21.15.173:0/663636342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0f5410 0x7f933c0f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f9330009fb0 tx=0x7f9330009b30).stop 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.588+0000 7f934304d700 1 -- 172.21.15.173:0/663636342 shutdown_connections 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.588+0000 7f934304d700 1 --2- 172.21.15.173:0/663636342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0f5410 0x7f933c0f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.588+0000 7f934304d700 1 -- 172.21.15.173:0/663636342 >> 172.21.15.173:0/663636342 conn(0x7f933c0f0b70 msgr2=0x7f933c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.588+0000 7f934304d700 1 -- 172.21.15.173:0/663636342 shutdown_connections 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.588+0000 7f934304d700 1 -- 172.21.15.173:0/663636342 wait complete. 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.589+0000 7f934304d700 1 Processor -- start 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.589+0000 7f934304d700 1 -- start start 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.589+0000 7f934304d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0f5410 0x7f933c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.589+0000 7f934304d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c060240 con 0x7f933c0f5410 2024-02-19T22:05:10.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.589+0000 7f9340de9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0f5410 0x7f933c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.589+0000 7f9340de9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0f5410 0x7f933c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:41414/0 (socket says 172.21.15.173:41414) 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.589+0000 7f9340de9700 1 -- 172.21.15.173:0/1515231651 learned_addr learned my addr 172.21.15.173:0/1515231651 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.589+0000 7f9340de9700 1 -- 172.21.15.173:0/1515231651 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9330005040 con 0x7f933c0f5410 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.590+0000 7f9340de9700 1 --2- 172.21.15.173:0/1515231651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0f5410 0x7f933c1094e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f9330005010 tx=0x7f933000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.590+0000 7f93397fa700 1 -- 172.21.15.173:0/1515231651 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9330009320 con 0x7f933c0f5410 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.590+0000 7f934304d700 1 -- 172.21.15.173:0/1515231651 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c060440 con 0x7f933c0f5410 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.590+0000 7f93397fa700 1 -- 172.21.15.173:0/1515231651 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9330013380 con 0x7f933c0f5410 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.590+0000 7f93397fa700 1 -- 172.21.15.173:0/1515231651 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9330014e50 con 0x7f933c0f5410 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.590+0000 7f934304d700 1 -- 172.21.15.173:0/1515231651 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c109b90 con 0x7f933c0f5410 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.591+0000 7f93397fa700 1 -- 172.21.15.173:0/1515231651 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93300263a0 con 0x7f933c0f5410 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.591+0000 7f93397fa700 1 --2- 172.21.15.173:0/1515231651 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f932c02ff10 0x7f932c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.591+0000 7f93397fa700 1 -- 172.21.15.173:0/1515231651 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9330040e70 con 0x7f933c0f5410 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.591+0000 7f934304d700 1 -- 172.21.15.173:0/1515231651 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c045ad0 con 0x7f933c0f5410 2024-02-19T22:05:10.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.591+0000 7f933bfff700 1 --2- 172.21.15.173:0/1515231651 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f932c02ff10 0x7f932c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:10.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.591+0000 7f933bfff700 1 --2- 172.21.15.173:0/1515231651 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f932c02ff10 0x7f932c0323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f93280096a0 tx=0x7f9328006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:10.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.596+0000 7f93397fa700 1 -- 172.21.15.173:0/1515231651 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9330012920 con 0x7f933c0f5410 2024-02-19T22:05:10.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.803+0000 7f934304d700 1 -- 172.21.15.173:0/1515231651 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f933c10a300 con 0x7f933c0f5410 2024-02-19T22:05:10.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.804+0000 7f93397fa700 1 -- 172.21.15.173:0/1515231651 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f933002b020 con 0x7f933c0f5410 2024-02-19T22:05:10.803 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:10.803 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:10.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.805+0000 7f934304d700 1 -- 172.21.15.173:0/1515231651 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f932c02ff10 msgr2=0x7f932c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:10.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.805+0000 7f934304d700 1 --2- 172.21.15.173:0/1515231651 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f932c02ff10 0x7f932c0323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f93280096a0 tx=0x7f9328006b40).stop 2024-02-19T22:05:10.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.805+0000 7f934304d700 1 -- 172.21.15.173:0/1515231651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0f5410 msgr2=0x7f933c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:10.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.805+0000 7f934304d700 1 --2- 172.21.15.173:0/1515231651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0f5410 0x7f933c1094e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f9330005010 tx=0x7f933000b040).stop 2024-02-19T22:05:10.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.805+0000 7f934304d700 1 -- 172.21.15.173:0/1515231651 shutdown_connections 2024-02-19T22:05:10.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.805+0000 7f934304d700 1 --2- 172.21.15.173:0/1515231651 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f932c02ff10 0x7f932c0323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:10.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.805+0000 7f934304d700 1 --2- 172.21.15.173:0/1515231651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0f5410 0x7f933c1094e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:10.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.805+0000 7f934304d700 1 -- 172.21.15.173:0/1515231651 >> 172.21.15.173:0/1515231651 conn(0x7f933c0f0b70 msgr2=0x7f933c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:10.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.805+0000 7f934304d700 1 -- 172.21.15.173:0/1515231651 shutdown_connections 2024-02-19T22:05:10.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:10.805+0000 7f934304d700 1 -- 172.21.15.173:0/1515231651 wait complete. 2024-02-19T22:05:10.806 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:11.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:10.804687+0000 mon.smithi143 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.173:0/1515231651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:12.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:11 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:11.336627+0000 mgr.smithi143.bissbc (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:12.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:12.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:14.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.440+0000 7f1072321700 1 -- 172.21.15.173:0/1246350571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c0f6510 msgr2=0x7f106c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:14.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.440+0000 7f1072321700 1 --2- 172.21.15.173:0/1246350571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c0f6510 0x7f106c0f6930 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f105c009fb0 tx=0x7f105c009b30).stop 2024-02-19T22:05:14.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.440+0000 7f1072321700 1 -- 172.21.15.173:0/1246350571 shutdown_connections 2024-02-19T22:05:14.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.440+0000 7f1072321700 1 --2- 172.21.15.173:0/1246350571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c0f6510 0x7f106c0f6930 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:14.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.440+0000 7f1072321700 1 -- 172.21.15.173:0/1246350571 >> 172.21.15.173:0/1246350571 conn(0x7f106c0f1c70 msgr2=0x7f106c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:14.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.440+0000 7f1072321700 1 -- 172.21.15.173:0/1246350571 shutdown_connections 2024-02-19T22:05:14.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.440+0000 7f1072321700 1 -- 172.21.15.173:0/1246350571 wait complete. 2024-02-19T22:05:14.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.441+0000 7f1072321700 1 Processor -- start 2024-02-19T22:05:14.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.441+0000 7f1072321700 1 -- start start 2024-02-19T22:05:14.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.441+0000 7f1072321700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c0f6510 0x7f106c108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:14.471 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.441+0000 7f1072321700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c108ef0 con 0x7f106c0f6510 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.441+0000 7f106bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c0f6510 0x7f106c108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.441+0000 7f106bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c0f6510 0x7f106c108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:36998/0 (socket says 172.21.15.173:36998) 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.441+0000 7f106bfff700 1 -- 172.21.15.173:0/1218141017 learned_addr learned my addr 172.21.15.173:0/1218141017 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.441+0000 7f106bfff700 1 -- 172.21.15.173:0/1218141017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c005040 con 0x7f106c0f6510 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.442+0000 7f106bfff700 1 --2- 172.21.15.173:0/1218141017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c0f6510 0x7f106c108a00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f105c005010 tx=0x7f105c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.442+0000 7f1068ff9700 1 -- 172.21.15.173:0/1218141017 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f105c009320 con 0x7f106c0f6510 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.442+0000 7f1068ff9700 1 -- 172.21.15.173:0/1218141017 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f105c013380 con 0x7f106c0f6510 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.442+0000 7f1072321700 1 -- 172.21.15.173:0/1218141017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c1090f0 con 0x7f106c0f6510 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.442+0000 7f1068ff9700 1 -- 172.21.15.173:0/1218141017 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f105c014e50 con 0x7f106c0f6510 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.442+0000 7f1072321700 1 -- 172.21.15.173:0/1218141017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c109580 con 0x7f106c0f6510 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.442+0000 7f1068ff9700 1 -- 172.21.15.173:0/1218141017 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f105c014420 con 0x7f106c0f6510 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.443+0000 7f1072321700 1 -- 172.21.15.173:0/1218141017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c045ad0 con 0x7f106c0f6510 2024-02-19T22:05:14.472 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.443+0000 7f1068ff9700 1 --2- 172.21.15.173:0/1218141017 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f105402ff10 0x7f10540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:14.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.443+0000 7f1068ff9700 1 -- 172.21.15.173:0/1218141017 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f105c040e70 con 0x7f106c0f6510 2024-02-19T22:05:14.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.443+0000 7f106b7fe700 1 --2- 172.21.15.173:0/1218141017 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f105402ff10 0x7f10540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:14.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.443+0000 7f106b7fe700 1 --2- 172.21.15.173:0/1218141017 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f105402ff10 0x7f10540323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f10600096a0 tx=0x7f1060006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:14.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.448+0000 7f1068ff9700 1 -- 172.21.15.173:0/1218141017 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f105c0095f0 con 0x7f106c0f6510 2024-02-19T22:05:14.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.654+0000 7f1072321700 1 -- 172.21.15.173:0/1218141017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f106c10a430 con 0x7f106c0f6510 2024-02-19T22:05:14.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.656+0000 7f1068ff9700 1 -- 172.21.15.173:0/1218141017 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f105c01db70 con 0x7f106c0f6510 2024-02-19T22:05:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:14.655 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:14.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.656+0000 7f1072321700 1 -- 172.21.15.173:0/1218141017 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f105402ff10 msgr2=0x7f10540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:14.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.656+0000 7f1072321700 1 --2- 172.21.15.173:0/1218141017 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f105402ff10 0x7f10540323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f10600096a0 tx=0x7f1060006b40).stop 2024-02-19T22:05:14.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.656+0000 7f1072321700 1 -- 172.21.15.173:0/1218141017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c0f6510 msgr2=0x7f106c108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:14.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.656+0000 7f1072321700 1 --2- 172.21.15.173:0/1218141017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c0f6510 0x7f106c108a00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f105c005010 tx=0x7f105c00b040).stop 2024-02-19T22:05:14.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.656+0000 7f1072321700 1 -- 172.21.15.173:0/1218141017 shutdown_connections 2024-02-19T22:05:14.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.656+0000 7f1072321700 1 --2- 172.21.15.173:0/1218141017 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f105402ff10 0x7f10540323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:14.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.656+0000 7f1072321700 1 --2- 172.21.15.173:0/1218141017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c0f6510 0x7f106c108a00 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:14.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.656+0000 7f1072321700 1 -- 172.21.15.173:0/1218141017 >> 172.21.15.173:0/1218141017 conn(0x7f106c0f1c70 msgr2=0x7f106c0f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:14.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.657+0000 7f1072321700 1 -- 172.21.15.173:0/1218141017 shutdown_connections 2024-02-19T22:05:14.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:14.657+0000 7f1072321700 1 -- 172.21.15.173:0/1218141017 wait complete. 2024-02-19T22:05:14.657 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:14.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:14 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:13.336920+0000 mgr.smithi143.bissbc (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:15.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:15 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:14.656087+0000 mon.smithi143 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.173:0/1218141017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:16.298 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:16.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:16.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:15.337230+0000 mgr.smithi143.bissbc (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:18.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.432+0000 7f96eb14b700 1 -- 172.21.15.173:0/1434313434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96e40f5410 msgr2=0x7f96e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:18.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.432+0000 7f96eb14b700 1 --2- 172.21.15.173:0/1434313434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96e40f5410 0x7f96e40f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f96d8009fb0 tx=0x7f96d8009b30).stop 2024-02-19T22:05:18.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.432+0000 7f96eb14b700 1 -- 172.21.15.173:0/1434313434 shutdown_connections 2024-02-19T22:05:18.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.432+0000 7f96eb14b700 1 --2- 172.21.15.173:0/1434313434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96e40f5410 0x7f96e40f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:18.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.432+0000 7f96eb14b700 1 -- 172.21.15.173:0/1434313434 >> 172.21.15.173:0/1434313434 conn(0x7f96e40f0b70 msgr2=0x7f96e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:18.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.432+0000 7f96eb14b700 1 -- 172.21.15.173:0/1434313434 shutdown_connections 2024-02-19T22:05:18.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.432+0000 7f96eb14b700 1 -- 172.21.15.173:0/1434313434 wait complete. 2024-02-19T22:05:18.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.433+0000 7f96eb14b700 1 Processor -- start 2024-02-19T22:05:18.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.433+0000 7f96eb14b700 1 -- start start 2024-02-19T22:05:18.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.433+0000 7f96eb14b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96e40f5410 0x7f96e41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:18.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.433+0000 7f96eb14b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4106cd0 con 0x7f96e40f5410 2024-02-19T22:05:18.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.434+0000 7f96e8ee7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96e40f5410 0x7f96e41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:18.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.434+0000 7f96e8ee7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96e40f5410 0x7f96e41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:37016/0 (socket says 172.21.15.173:37016) 2024-02-19T22:05:18.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.434+0000 7f96e8ee7700 1 -- 172.21.15.173:0/342348750 learned_addr learned my addr 172.21.15.173:0/342348750 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:18.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.434+0000 7f96e8ee7700 1 -- 172.21.15.173:0/342348750 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d8005040 con 0x7f96e40f5410 2024-02-19T22:05:18.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.434+0000 7f96e8ee7700 1 --2- 172.21.15.173:0/342348750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96e40f5410 0x7f96e41067e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f96d8005010 tx=0x7f96d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:18.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.434+0000 7f96e17fa700 1 -- 172.21.15.173:0/342348750 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96d8009320 con 0x7f96e40f5410 2024-02-19T22:05:18.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.434+0000 7f96eb14b700 1 -- 172.21.15.173:0/342348750 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e4106ed0 con 0x7f96e40f5410 2024-02-19T22:05:18.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.434+0000 7f96e17fa700 1 -- 172.21.15.173:0/342348750 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96d8013380 con 0x7f96e40f5410 2024-02-19T22:05:18.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.434+0000 7f96e17fa700 1 -- 172.21.15.173:0/342348750 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96d8014e50 con 0x7f96e40f5410 2024-02-19T22:05:18.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.434+0000 7f96eb14b700 1 -- 172.21.15.173:0/342348750 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e4107360 con 0x7f96e40f5410 2024-02-19T22:05:18.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.435+0000 7f96e17fa700 1 -- 172.21.15.173:0/342348750 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96d80263a0 con 0x7f96e40f5410 2024-02-19T22:05:18.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.435+0000 7f96e17fa700 1 --2- 172.21.15.173:0/342348750 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96d402ff10 0x7f96d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:18.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.435+0000 7f96e17fa700 1 -- 172.21.15.173:0/342348750 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96d8040f10 con 0x7f96e40f5410 2024-02-19T22:05:18.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.435+0000 7f96eb14b700 1 -- 172.21.15.173:0/342348750 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e4045ad0 con 0x7f96e40f5410 2024-02-19T22:05:18.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.436+0000 7f96e3fff700 1 --2- 172.21.15.173:0/342348750 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96d402ff10 0x7f96d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:18.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.436+0000 7f96e3fff700 1 --2- 172.21.15.173:0/342348750 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96d402ff10 0x7f96d40323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f96d00096a0 tx=0x7f96d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:18.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.441+0000 7f96e17fa700 1 -- 172.21.15.173:0/342348750 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96d8012920 con 0x7f96e40f5410 2024-02-19T22:05:18.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.650+0000 7f96eb14b700 1 -- 172.21.15.173:0/342348750 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96e4108080 con 0x7f96e40f5410 2024-02-19T22:05:18.650 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.651+0000 7f96e17fa700 1 -- 172.21.15.173:0/342348750 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f96d802b020 con 0x7f96e40f5410 2024-02-19T22:05:18.651 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:18.651 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:18.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.652+0000 7f96eb14b700 1 -- 172.21.15.173:0/342348750 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96d402ff10 msgr2=0x7f96d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:18.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.652+0000 7f96eb14b700 1 --2- 172.21.15.173:0/342348750 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96d402ff10 0x7f96d40323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f96d00096a0 tx=0x7f96d0006b40).stop 2024-02-19T22:05:18.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.652+0000 7f96eb14b700 1 -- 172.21.15.173:0/342348750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96e40f5410 msgr2=0x7f96e41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:18.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.652+0000 7f96eb14b700 1 --2- 172.21.15.173:0/342348750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96e40f5410 0x7f96e41067e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f96d8005010 tx=0x7f96d800b040).stop 2024-02-19T22:05:18.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.653+0000 7f96eb14b700 1 -- 172.21.15.173:0/342348750 shutdown_connections 2024-02-19T22:05:18.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.653+0000 7f96eb14b700 1 --2- 172.21.15.173:0/342348750 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f96d402ff10 0x7f96d40323d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:18.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.653+0000 7f96eb14b700 1 --2- 172.21.15.173:0/342348750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96e40f5410 0x7f96e41067e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:18.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.653+0000 7f96eb14b700 1 -- 172.21.15.173:0/342348750 >> 172.21.15.173:0/342348750 conn(0x7f96e40f0b70 msgr2=0x7f96e405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:18.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.653+0000 7f96eb14b700 1 -- 172.21.15.173:0/342348750 shutdown_connections 2024-02-19T22:05:18.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:18.653+0000 7f96eb14b700 1 -- 172.21.15.173:0/342348750 wait complete. 2024-02-19T22:05:18.653 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:18.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:17.337493+0000 mgr.smithi143.bissbc (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:19.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:19 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:18.652033+0000 mon.smithi143 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.173:0/342348750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:20.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:20.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:19.337882+0000 mgr.smithi143.bissbc (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:20.574134+0000 mon.smithi143 (mon.0) 313 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T22:05:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:20.574331+0000 mon.smithi143 (mon.0) 314 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:05:22.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.489+0000 7fec7e118700 1 -- 172.21.15.173:0/532601708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780f62a0 msgr2=0x7fec780f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:22.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.489+0000 7fec7e118700 1 --2- 172.21.15.173:0/532601708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780f62a0 0x7fec780f66c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fec74009fb0 tx=0x7fec74009b30).stop 2024-02-19T22:05:22.489 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.489+0000 7fec7e118700 1 -- 172.21.15.173:0/532601708 shutdown_connections 2024-02-19T22:05:22.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.489+0000 7fec7e118700 1 --2- 172.21.15.173:0/532601708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780f62a0 0x7fec780f66c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:22.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.489+0000 7fec7e118700 1 -- 172.21.15.173:0/532601708 >> 172.21.15.173:0/532601708 conn(0x7fec780f1aa0 msgr2=0x7fec780f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:22.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.489+0000 7fec7e118700 1 -- 172.21.15.173:0/532601708 shutdown_connections 2024-02-19T22:05:22.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.489+0000 7fec7e118700 1 -- 172.21.15.173:0/532601708 wait complete. 2024-02-19T22:05:22.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.490+0000 7fec7e118700 1 Processor -- start 2024-02-19T22:05:22.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.490+0000 7fec7e118700 1 -- start start 2024-02-19T22:05:22.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.490+0000 7fec7e118700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780f62a0 0x7fec781065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:22.490 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.490+0000 7fec7e118700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec78106ac0 con 0x7fec780f62a0 2024-02-19T22:05:22.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.490+0000 7fec7d116700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780f62a0 0x7fec781065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:22.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.490+0000 7fec7d116700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780f62a0 0x7fec781065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:48538/0 (socket says 172.21.15.173:48538) 2024-02-19T22:05:22.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.490+0000 7fec7d116700 1 -- 172.21.15.173:0/3321359098 learned_addr learned my addr 172.21.15.173:0/3321359098 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:22.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.490+0000 7fec7d116700 1 -- 172.21.15.173:0/3321359098 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec74005040 con 0x7fec780f62a0 2024-02-19T22:05:22.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.491+0000 7fec7d116700 1 --2- 172.21.15.173:0/3321359098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780f62a0 0x7fec781065d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fec740089f0 tx=0x7fec7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:22.492 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.491+0000 7fec6dffb700 1 -- 172.21.15.173:0/3321359098 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec74009320 con 0x7fec780f62a0 2024-02-19T22:05:22.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.491+0000 7fec7e118700 1 -- 172.21.15.173:0/3321359098 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec78106cc0 con 0x7fec780f62a0 2024-02-19T22:05:22.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.491+0000 7fec6dffb700 1 -- 172.21.15.173:0/3321359098 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec74013380 con 0x7fec780f62a0 2024-02-19T22:05:22.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.491+0000 7fec6dffb700 1 -- 172.21.15.173:0/3321359098 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec74014e20 con 0x7fec780f62a0 2024-02-19T22:05:22.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.491+0000 7fec7e118700 1 -- 172.21.15.173:0/3321359098 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec78107150 con 0x7fec780f62a0 2024-02-19T22:05:22.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.492+0000 7fec6dffb700 1 -- 172.21.15.173:0/3321359098 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec740095f0 con 0x7fec780f62a0 2024-02-19T22:05:22.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.492+0000 7fec7e118700 1 -- 172.21.15.173:0/3321359098 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec78045ad0 con 0x7fec780f62a0 2024-02-19T22:05:22.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.492+0000 7fec6dffb700 1 --2- 172.21.15.173:0/3321359098 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fec6402ff10 0x7fec640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:22.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.492+0000 7fec6dffb700 1 -- 172.21.15.173:0/3321359098 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec74040e90 con 0x7fec780f62a0 2024-02-19T22:05:22.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.492+0000 7fec7c915700 1 --2- 172.21.15.173:0/3321359098 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fec6402ff10 0x7fec640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:22.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.493+0000 7fec7c915700 1 --2- 172.21.15.173:0/3321359098 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fec6402ff10 0x7fec640323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fec680096a0 tx=0x7fec68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:22.496 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.497+0000 7fec6dffb700 1 -- 172.21.15.173:0/3321359098 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec74014420 con 0x7fec780f62a0 2024-02-19T22:05:22.704 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.704+0000 7fec7e118700 1 -- 172.21.15.173:0/3321359098 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec78108000 con 0x7fec780f62a0 2024-02-19T22:05:22.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.705+0000 7fec6dffb700 1 -- 172.21.15.173:0/3321359098 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fec74010230 con 0x7fec780f62a0 2024-02-19T22:05:22.705 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:22.705 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:22.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.707+0000 7fec7e118700 1 -- 172.21.15.173:0/3321359098 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fec6402ff10 msgr2=0x7fec640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:22.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.707+0000 7fec7e118700 1 --2- 172.21.15.173:0/3321359098 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fec6402ff10 0x7fec640323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fec680096a0 tx=0x7fec68006b40).stop 2024-02-19T22:05:22.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.707+0000 7fec7e118700 1 -- 172.21.15.173:0/3321359098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780f62a0 msgr2=0x7fec781065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:22.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.707+0000 7fec7e118700 1 --2- 172.21.15.173:0/3321359098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780f62a0 0x7fec781065d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fec740089f0 tx=0x7fec7400b040).stop 2024-02-19T22:05:22.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.707+0000 7fec7e118700 1 -- 172.21.15.173:0/3321359098 shutdown_connections 2024-02-19T22:05:22.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.707+0000 7fec7e118700 1 --2- 172.21.15.173:0/3321359098 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fec6402ff10 0x7fec640323d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:22.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.707+0000 7fec7e118700 1 --2- 172.21.15.173:0/3321359098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec780f62a0 0x7fec781065d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:22.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.707+0000 7fec7e118700 1 -- 172.21.15.173:0/3321359098 >> 172.21.15.173:0/3321359098 conn(0x7fec780f1aa0 msgr2=0x7fec780f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:22.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.707+0000 7fec7e118700 1 -- 172.21.15.173:0/3321359098 shutdown_connections 2024-02-19T22:05:22.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:22.707+0000 7fec7e118700 1 -- 172.21.15.173:0/3321359098 wait complete. 2024-02-19T22:05:22.708 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:21.338131+0000 mgr.smithi143.bissbc (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:23.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:22.706363+0000 mon.smithi143 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.173:0/3321359098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:23.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:23.338444+0000 mgr.smithi143.bissbc (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:24.426 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:24.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:26.624 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.623+0000 7f61fc324700 1 -- 172.21.15.173:0/945379432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61f40f6510 msgr2=0x7f61f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:26.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.623+0000 7f61fc324700 1 --2- 172.21.15.173:0/945379432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61f40f6510 0x7f61f40f6930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f61e4009fb0 tx=0x7f61e4009b30).stop 2024-02-19T22:05:26.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.623+0000 7f61fc324700 1 -- 172.21.15.173:0/945379432 shutdown_connections 2024-02-19T22:05:26.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.623+0000 7f61fc324700 1 --2- 172.21.15.173:0/945379432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61f40f6510 0x7f61f40f6930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:26.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.623+0000 7f61fc324700 1 -- 172.21.15.173:0/945379432 >> 172.21.15.173:0/945379432 conn(0x7f61f40f1c70 msgr2=0x7f61f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:26.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.623+0000 7f61fc324700 1 -- 172.21.15.173:0/945379432 shutdown_connections 2024-02-19T22:05:26.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.623+0000 7f61fc324700 1 -- 172.21.15.173:0/945379432 wait complete. 2024-02-19T22:05:26.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.624+0000 7f61fc324700 1 Processor -- start 2024-02-19T22:05:26.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.624+0000 7f61fc324700 1 -- start start 2024-02-19T22:05:26.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.624+0000 7f61fc324700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61f40f6510 0x7f61f406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:26.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.624+0000 7f61fc324700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f4069e20 con 0x7f61f40f6510 2024-02-19T22:05:26.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.624+0000 7f61fa0c0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61f40f6510 0x7f61f406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:26.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.624+0000 7f61fa0c0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61f40f6510 0x7f61f406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:48560/0 (socket says 172.21.15.173:48560) 2024-02-19T22:05:26.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.624+0000 7f61fa0c0700 1 -- 172.21.15.173:0/3724687984 learned_addr learned my addr 172.21.15.173:0/3724687984 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:26.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.624+0000 7f61fa0c0700 1 -- 172.21.15.173:0/3724687984 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e4005040 con 0x7f61f40f6510 2024-02-19T22:05:26.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.625+0000 7f61fa0c0700 1 --2- 172.21.15.173:0/3724687984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61f40f6510 0x7f61f406b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f61e4005010 tx=0x7f61e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:26.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.625+0000 7f61eaffd700 1 -- 172.21.15.173:0/3724687984 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61e4009320 con 0x7f61f40f6510 2024-02-19T22:05:26.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.625+0000 7f61fc324700 1 -- 172.21.15.173:0/3724687984 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f4069fc0 con 0x7f61f40f6510 2024-02-19T22:05:26.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.625+0000 7f61eaffd700 1 -- 172.21.15.173:0/3724687984 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61e4013380 con 0x7f61f40f6510 2024-02-19T22:05:26.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.625+0000 7f61eaffd700 1 -- 172.21.15.173:0/3724687984 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61e4014e50 con 0x7f61f40f6510 2024-02-19T22:05:26.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.625+0000 7f61fc324700 1 -- 172.21.15.173:0/3724687984 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f406a450 con 0x7f61f40f6510 2024-02-19T22:05:26.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.625+0000 7f61eaffd700 1 -- 172.21.15.173:0/3724687984 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61e40263a0 con 0x7f61f40f6510 2024-02-19T22:05:26.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.626+0000 7f61eaffd700 1 --2- 172.21.15.173:0/3724687984 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f61e002ff10 0x7f61e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:26.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.626+0000 7f61eaffd700 1 -- 172.21.15.173:0/3724687984 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61e4040ec0 con 0x7f61f40f6510 2024-02-19T22:05:26.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.626+0000 7f61fc324700 1 -- 172.21.15.173:0/3724687984 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f4059100 con 0x7f61f40f6510 2024-02-19T22:05:26.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.626+0000 7f61f98bf700 1 --2- 172.21.15.173:0/3724687984 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f61e002ff10 0x7f61e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:26.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.626+0000 7f61f98bf700 1 --2- 172.21.15.173:0/3724687984 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f61e002ff10 0x7f61e00323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f61f00096a0 tx=0x7f61f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:26.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.631+0000 7f61eaffd700 1 -- 172.21.15.173:0/3724687984 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61e4012920 con 0x7f61f40f6510 2024-02-19T22:05:26.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:25.338688+0000 mgr.smithi143.bissbc (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:26.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.838+0000 7f61fc324700 1 -- 172.21.15.173:0/3724687984 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61f4045ad0 con 0x7f61f40f6510 2024-02-19T22:05:26.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.838+0000 7f61eaffd700 1 -- 172.21.15.173:0/3724687984 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f61e402b020 con 0x7f61f40f6510 2024-02-19T22:05:26.838 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:26.838 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:26.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.840+0000 7f61fc324700 1 -- 172.21.15.173:0/3724687984 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f61e002ff10 msgr2=0x7f61e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:26.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.840+0000 7f61fc324700 1 --2- 172.21.15.173:0/3724687984 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f61e002ff10 0x7f61e00323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f61f00096a0 tx=0x7f61f0006b40).stop 2024-02-19T22:05:26.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.840+0000 7f61fc324700 1 -- 172.21.15.173:0/3724687984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61f40f6510 msgr2=0x7f61f406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:26.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.840+0000 7f61fc324700 1 --2- 172.21.15.173:0/3724687984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61f40f6510 0x7f61f406b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f61e4005010 tx=0x7f61e400b040).stop 2024-02-19T22:05:26.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.840+0000 7f61fc324700 1 -- 172.21.15.173:0/3724687984 shutdown_connections 2024-02-19T22:05:26.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.840+0000 7f61fc324700 1 --2- 172.21.15.173:0/3724687984 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f61e002ff10 0x7f61e00323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:26.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.840+0000 7f61fc324700 1 --2- 172.21.15.173:0/3724687984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61f40f6510 0x7f61f406b9a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:26.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.840+0000 7f61fc324700 1 -- 172.21.15.173:0/3724687984 >> 172.21.15.173:0/3724687984 conn(0x7f61f40f1c70 msgr2=0x7f61f40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:26.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.840+0000 7f61fc324700 1 -- 172.21.15.173:0/3724687984 shutdown_connections 2024-02-19T22:05:26.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:26.840+0000 7f61fc324700 1 -- 172.21.15.173:0/3724687984 wait complete. 2024-02-19T22:05:26.840 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:27.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:27 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:26.839426+0000 mon.smithi143 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.173:0/3724687984' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:28.494 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:28.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:27.339018+0000 mgr.smithi143.bissbc (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:30.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.645+0000 7f2c48b94700 1 -- 172.21.15.173:0/2546028677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c440f5410 msgr2=0x7f2c440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:30.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.645+0000 7f2c48b94700 1 --2- 172.21.15.173:0/2546028677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c440f5410 0x7f2c440f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f2c2c009fb0 tx=0x7f2c2c009b30).stop 2024-02-19T22:05:30.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.645+0000 7f2c48b94700 1 -- 172.21.15.173:0/2546028677 shutdown_connections 2024-02-19T22:05:30.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.645+0000 7f2c48b94700 1 --2- 172.21.15.173:0/2546028677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c440f5410 0x7f2c440f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:30.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.645+0000 7f2c48b94700 1 -- 172.21.15.173:0/2546028677 >> 172.21.15.173:0/2546028677 conn(0x7f2c440f0b70 msgr2=0x7f2c440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:30.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.645+0000 7f2c48b94700 1 -- 172.21.15.173:0/2546028677 shutdown_connections 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.645+0000 7f2c48b94700 1 -- 172.21.15.173:0/2546028677 wait complete. 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.646+0000 7f2c48b94700 1 Processor -- start 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.646+0000 7f2c48b94700 1 -- start start 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.646+0000 7f2c48b94700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c440f5410 0x7f2c441056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.646+0000 7f2c48b94700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c44105bd0 con 0x7f2c440f5410 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.646+0000 7f2c4259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c440f5410 0x7f2c441056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.646+0000 7f2c4259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c440f5410 0x7f2c441056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:48584/0 (socket says 172.21.15.173:48584) 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.646+0000 7f2c4259c700 1 -- 172.21.15.173:0/3249447561 learned_addr learned my addr 172.21.15.173:0/3249447561 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.647+0000 7f2c4259c700 1 -- 172.21.15.173:0/3249447561 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c005040 con 0x7f2c440f5410 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.647+0000 7f2c4259c700 1 --2- 172.21.15.173:0/3249447561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c440f5410 0x7f2c441056e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f2c2c005010 tx=0x7f2c2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.647+0000 7f2c3affd700 1 -- 172.21.15.173:0/3249447561 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c2c009320 con 0x7f2c440f5410 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.647+0000 7f2c48b94700 1 -- 172.21.15.173:0/3249447561 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c44105dd0 con 0x7f2c440f5410 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.647+0000 7f2c3affd700 1 -- 172.21.15.173:0/3249447561 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c2c013380 con 0x7f2c440f5410 2024-02-19T22:05:30.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.647+0000 7f2c3affd700 1 -- 172.21.15.173:0/3249447561 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c2c014e50 con 0x7f2c440f5410 2024-02-19T22:05:30.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.647+0000 7f2c48b94700 1 -- 172.21.15.173:0/3249447561 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c44106260 con 0x7f2c440f5410 2024-02-19T22:05:30.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.648+0000 7f2c3affd700 1 -- 172.21.15.173:0/3249447561 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c2c0263a0 con 0x7f2c440f5410 2024-02-19T22:05:30.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.648+0000 7f2c3affd700 1 --2- 172.21.15.173:0/3249447561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2c3002ff10 0x7f2c300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:30.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.648+0000 7f2c3affd700 1 -- 172.21.15.173:0/3249447561 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c2c040f10 con 0x7f2c440f5410 2024-02-19T22:05:30.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.648+0000 7f2c48b94700 1 -- 172.21.15.173:0/3249447561 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c44045ad0 con 0x7f2c440f5410 2024-02-19T22:05:30.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.648+0000 7f2c41d9b700 1 --2- 172.21.15.173:0/3249447561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2c3002ff10 0x7f2c300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:30.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.648+0000 7f2c41d9b700 1 --2- 172.21.15.173:0/3249447561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2c3002ff10 0x7f2c300323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f2c340096a0 tx=0x7f2c34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:30.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.653+0000 7f2c3affd700 1 -- 172.21.15.173:0/3249447561 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c2c012920 con 0x7f2c440f5410 2024-02-19T22:05:30.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.861+0000 7f2c48b94700 1 -- 172.21.15.173:0/3249447561 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c44106fb0 con 0x7f2c440f5410 2024-02-19T22:05:30.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.862+0000 7f2c3affd700 1 -- 172.21.15.173:0/3249447561 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2c2c02b020 con 0x7f2c440f5410 2024-02-19T22:05:30.862 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:30.862 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:30.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.864+0000 7f2c48b94700 1 -- 172.21.15.173:0/3249447561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2c3002ff10 msgr2=0x7f2c300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:30.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.864+0000 7f2c48b94700 1 --2- 172.21.15.173:0/3249447561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2c3002ff10 0x7f2c300323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f2c340096a0 tx=0x7f2c34006b40).stop 2024-02-19T22:05:30.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.864+0000 7f2c48b94700 1 -- 172.21.15.173:0/3249447561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c440f5410 msgr2=0x7f2c441056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:30.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.864+0000 7f2c48b94700 1 --2- 172.21.15.173:0/3249447561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c440f5410 0x7f2c441056e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f2c2c005010 tx=0x7f2c2c00b040).stop 2024-02-19T22:05:30.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.864+0000 7f2c48b94700 1 -- 172.21.15.173:0/3249447561 shutdown_connections 2024-02-19T22:05:30.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.864+0000 7f2c48b94700 1 --2- 172.21.15.173:0/3249447561 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f2c3002ff10 0x7f2c300323d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:30.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.864+0000 7f2c48b94700 1 --2- 172.21.15.173:0/3249447561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c440f5410 0x7f2c441056e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:30.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.864+0000 7f2c48b94700 1 -- 172.21.15.173:0/3249447561 >> 172.21.15.173:0/3249447561 conn(0x7f2c440f0b70 msgr2=0x7f2c440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:30.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.864+0000 7f2c48b94700 1 -- 172.21.15.173:0/3249447561 shutdown_connections 2024-02-19T22:05:30.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:30.864+0000 7f2c48b94700 1 -- 172.21.15.173:0/3249447561 wait complete. 2024-02-19T22:05:30.865 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:29.339259+0000 mgr.smithi143.bissbc (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:31.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:31 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:30.863388+0000 mon.smithi143 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.173:0/3249447561' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:32.562 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:32.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:32.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:31.339620+0000 mgr.smithi143.bissbc (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:34.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.614+0000 7f9d44cca700 1 -- 172.21.15.173:0/2505249122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d400601e0 msgr2=0x7f9d400f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:34.615 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.614+0000 7f9d44cca700 1 --2- 172.21.15.173:0/2505249122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d400601e0 0x7f9d400f8f20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f9d30009fb0 tx=0x7f9d30009b30).stop 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.614+0000 7f9d44cca700 1 -- 172.21.15.173:0/2505249122 shutdown_connections 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.614+0000 7f9d44cca700 1 --2- 172.21.15.173:0/2505249122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d400601e0 0x7f9d400f8f20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.614+0000 7f9d44cca700 1 -- 172.21.15.173:0/2505249122 >> 172.21.15.173:0/2505249122 conn(0x7f9d400f09a0 msgr2=0x7f9d400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.614+0000 7f9d44cca700 1 -- 172.21.15.173:0/2505249122 shutdown_connections 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.614+0000 7f9d44cca700 1 -- 172.21.15.173:0/2505249122 wait complete. 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.614+0000 7f9d44cca700 1 Processor -- start 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.615+0000 7f9d44cca700 1 -- start start 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.615+0000 7f9d44cca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d400601e0 0x7f9d401087f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.615+0000 7f9d44cca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d40108ce0 con 0x7f9d400601e0 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.615+0000 7f9d3f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d400601e0 0x7f9d401087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.615+0000 7f9d3f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d400601e0 0x7f9d401087f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:58858/0 (socket says 172.21.15.173:58858) 2024-02-19T22:05:34.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.615+0000 7f9d3f7fe700 1 -- 172.21.15.173:0/3690229145 learned_addr learned my addr 172.21.15.173:0/3690229145 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:34.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.615+0000 7f9d3f7fe700 1 -- 172.21.15.173:0/3690229145 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d30005040 con 0x7f9d400601e0 2024-02-19T22:05:34.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.615+0000 7f9d3f7fe700 1 --2- 172.21.15.173:0/3690229145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d400601e0 0x7f9d401087f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f9d300089f0 tx=0x7f9d3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:34.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.616+0000 7f9d27fff700 1 -- 172.21.15.173:0/3690229145 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d30009320 con 0x7f9d400601e0 2024-02-19T22:05:34.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.616+0000 7f9d44cca700 1 -- 172.21.15.173:0/3690229145 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d40108ee0 con 0x7f9d400601e0 2024-02-19T22:05:34.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.616+0000 7f9d27fff700 1 -- 172.21.15.173:0/3690229145 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d30013380 con 0x7f9d400601e0 2024-02-19T22:05:34.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.616+0000 7f9d27fff700 1 -- 172.21.15.173:0/3690229145 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d3001ed80 con 0x7f9d400601e0 2024-02-19T22:05:34.617 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.616+0000 7f9d44cca700 1 -- 172.21.15.173:0/3690229145 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d40109370 con 0x7f9d400601e0 2024-02-19T22:05:34.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.616+0000 7f9d27fff700 1 -- 172.21.15.173:0/3690229145 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d3001e420 con 0x7f9d400601e0 2024-02-19T22:05:34.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.617+0000 7f9d44cca700 1 -- 172.21.15.173:0/3690229145 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d40045ad0 con 0x7f9d400601e0 2024-02-19T22:05:34.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.617+0000 7f9d27fff700 1 --2- 172.21.15.173:0/3690229145 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d2802ff60 0x7f9d28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:34.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.617+0000 7f9d27fff700 1 -- 172.21.15.173:0/3690229145 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d30041120 con 0x7f9d400601e0 2024-02-19T22:05:34.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.617+0000 7f9d3effd700 1 --2- 172.21.15.173:0/3690229145 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d2802ff60 0x7f9d28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:34.618 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.617+0000 7f9d3effd700 1 --2- 172.21.15.173:0/3690229145 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d2802ff60 0x7f9d28032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f9d340096a0 tx=0x7f9d34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:34.621 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.622+0000 7f9d27fff700 1 -- 172.21.15.173:0/3690229145 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d3001cb10 con 0x7f9d400601e0 2024-02-19T22:05:34.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.830+0000 7f9d44cca700 1 -- 172.21.15.173:0/3690229145 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d4010a1c0 con 0x7f9d400601e0 2024-02-19T22:05:34.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.831+0000 7f9d27fff700 1 -- 172.21.15.173:0/3690229145 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d3002c2d0 con 0x7f9d400601e0 2024-02-19T22:05:34.830 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:34.831 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:34.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.833+0000 7f9d44cca700 1 -- 172.21.15.173:0/3690229145 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d2802ff60 msgr2=0x7f9d28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:34.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.833+0000 7f9d44cca700 1 --2- 172.21.15.173:0/3690229145 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d2802ff60 0x7f9d28032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f9d340096a0 tx=0x7f9d34006b40).stop 2024-02-19T22:05:34.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.833+0000 7f9d44cca700 1 -- 172.21.15.173:0/3690229145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d400601e0 msgr2=0x7f9d401087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:34.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.833+0000 7f9d44cca700 1 --2- 172.21.15.173:0/3690229145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d400601e0 0x7f9d401087f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f9d300089f0 tx=0x7f9d3000b040).stop 2024-02-19T22:05:34.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.833+0000 7f9d44cca700 1 -- 172.21.15.173:0/3690229145 shutdown_connections 2024-02-19T22:05:34.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.833+0000 7f9d44cca700 1 --2- 172.21.15.173:0/3690229145 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f9d2802ff60 0x7f9d28032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:34.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.833+0000 7f9d44cca700 1 --2- 172.21.15.173:0/3690229145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d400601e0 0x7f9d401087f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:34.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.833+0000 7f9d44cca700 1 -- 172.21.15.173:0/3690229145 >> 172.21.15.173:0/3690229145 conn(0x7f9d400f09a0 msgr2=0x7f9d400f1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:34.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.833+0000 7f9d44cca700 1 -- 172.21.15.173:0/3690229145 shutdown_connections 2024-02-19T22:05:34.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:34.833+0000 7f9d44cca700 1 -- 172.21.15.173:0/3690229145 wait complete. 2024-02-19T22:05:34.833 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:33.339985+0000 mgr.smithi143.bissbc (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:35.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:34.831936+0000 mon.smithi143 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.173:0/3690229145' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:36.583 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:36.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:35.340369+0000 mgr.smithi143.bissbc (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:38.785 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.785+0000 7ff9c1389700 1 -- 172.21.15.173:0/699371758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc0f51a0 msgr2=0x7ff9bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:38.785 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.785+0000 7ff9c1389700 1 --2- 172.21.15.173:0/699371758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc0f51a0 0x7ff9bc0f55c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7ff9ac009fb0 tx=0x7ff9ac009b30).stop 2024-02-19T22:05:38.786 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.785+0000 7ff9c1389700 1 -- 172.21.15.173:0/699371758 shutdown_connections 2024-02-19T22:05:38.786 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.785+0000 7ff9c1389700 1 --2- 172.21.15.173:0/699371758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc0f51a0 0x7ff9bc0f55c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:38.786 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.785+0000 7ff9c1389700 1 -- 172.21.15.173:0/699371758 >> 172.21.15.173:0/699371758 conn(0x7ff9bc0f09a0 msgr2=0x7ff9bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:38.786 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.785+0000 7ff9c1389700 1 -- 172.21.15.173:0/699371758 shutdown_connections 2024-02-19T22:05:38.786 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.785+0000 7ff9c1389700 1 -- 172.21.15.173:0/699371758 wait complete. 2024-02-19T22:05:38.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.785+0000 7ff9c1389700 1 Processor -- start 2024-02-19T22:05:38.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.786+0000 7ff9c1389700 1 -- start start 2024-02-19T22:05:38.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.786+0000 7ff9c1389700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc0f51a0 0x7ff9bc1092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:38.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.786+0000 7ff9c1389700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9bc060220 con 0x7ff9bc0f51a0 2024-02-19T22:05:38.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.786+0000 7ff9bbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc0f51a0 0x7ff9bc1092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:38.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.786+0000 7ff9bbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc0f51a0 0x7ff9bc1092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:58872/0 (socket says 172.21.15.173:58872) 2024-02-19T22:05:38.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.786+0000 7ff9bbfff700 1 -- 172.21.15.173:0/1631379325 learned_addr learned my addr 172.21.15.173:0/1631379325 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:38.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.786+0000 7ff9bbfff700 1 -- 172.21.15.173:0/1631379325 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac005040 con 0x7ff9bc0f51a0 2024-02-19T22:05:38.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.787+0000 7ff9bbfff700 1 --2- 172.21.15.173:0/1631379325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc0f51a0 0x7ff9bc1092f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff9ac0089f0 tx=0x7ff9ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:38.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.787+0000 7ff9b8ff9700 1 -- 172.21.15.173:0/1631379325 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9ac009320 con 0x7ff9bc0f51a0 2024-02-19T22:05:38.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.787+0000 7ff9c1389700 1 -- 172.21.15.173:0/1631379325 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9bc060420 con 0x7ff9bc0f51a0 2024-02-19T22:05:38.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.787+0000 7ff9b8ff9700 1 -- 172.21.15.173:0/1631379325 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9ac013380 con 0x7ff9bc0f51a0 2024-02-19T22:05:38.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.787+0000 7ff9b8ff9700 1 -- 172.21.15.173:0/1631379325 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9ac014e20 con 0x7ff9bc0f51a0 2024-02-19T22:05:38.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.787+0000 7ff9c1389700 1 -- 172.21.15.173:0/1631379325 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9bc1099a0 con 0x7ff9bc0f51a0 2024-02-19T22:05:38.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.788+0000 7ff9c1389700 1 -- 172.21.15.173:0/1631379325 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9bc059100 con 0x7ff9bc0f51a0 2024-02-19T22:05:38.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.788+0000 7ff9b8ff9700 1 -- 172.21.15.173:0/1631379325 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9ac0095f0 con 0x7ff9bc0f51a0 2024-02-19T22:05:38.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.788+0000 7ff9b8ff9700 1 --2- 172.21.15.173:0/1631379325 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9a402ff60 0x7ff9a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:38.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.788+0000 7ff9b8ff9700 1 -- 172.21.15.173:0/1631379325 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9ac01b920 con 0x7ff9bc0f51a0 2024-02-19T22:05:38.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.788+0000 7ff9bb7fe700 1 --2- 172.21.15.173:0/1631379325 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9a402ff60 0x7ff9a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:38.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.789+0000 7ff9bb7fe700 1 --2- 172.21.15.173:0/1631379325 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9a402ff60 0x7ff9a4032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff9b00096a0 tx=0x7ff9b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:38.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.793+0000 7ff9b8ff9700 1 -- 172.21.15.173:0/1631379325 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9ac014420 con 0x7ff9bc0f51a0 2024-02-19T22:05:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:37.340707+0000 mgr.smithi143.bissbc (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:38.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.997+0000 7ff9c1389700 1 -- 172.21.15.173:0/1631379325 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9bc0fad30 con 0x7ff9bc0f51a0 2024-02-19T22:05:39.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.998+0000 7ff9b8ff9700 1 -- 172.21.15.173:0/1631379325 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff9ac01b740 con 0x7ff9bc0f51a0 2024-02-19T22:05:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:39.010 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:39.011 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.999+0000 7ff9c1389700 1 -- 172.21.15.173:0/1631379325 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9a402ff60 msgr2=0x7ff9a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:39.011 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.999+0000 7ff9c1389700 1 --2- 172.21.15.173:0/1631379325 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9a402ff60 0x7ff9a4032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff9b00096a0 tx=0x7ff9b0006b40).stop 2024-02-19T22:05:39.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.999+0000 7ff9c1389700 1 -- 172.21.15.173:0/1631379325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc0f51a0 msgr2=0x7ff9bc1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:39.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.999+0000 7ff9c1389700 1 --2- 172.21.15.173:0/1631379325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc0f51a0 0x7ff9bc1092f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff9ac0089f0 tx=0x7ff9ac00b040).stop 2024-02-19T22:05:39.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.999+0000 7ff9c1389700 1 -- 172.21.15.173:0/1631379325 shutdown_connections 2024-02-19T22:05:39.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.999+0000 7ff9c1389700 1 --2- 172.21.15.173:0/1631379325 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff9a402ff60 0x7ff9a4032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:39.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.999+0000 7ff9c1389700 1 --2- 172.21.15.173:0/1631379325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9bc0f51a0 0x7ff9bc1092f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:39.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:38.999+0000 7ff9c1389700 1 -- 172.21.15.173:0/1631379325 >> 172.21.15.173:0/1631379325 conn(0x7ff9bc0f09a0 msgr2=0x7ff9bc0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:39.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:39.000+0000 7ff9c1389700 1 -- 172.21.15.173:0/1631379325 shutdown_connections 2024-02-19T22:05:39.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:39.000+0000 7ff9c1389700 1 -- 172.21.15.173:0/1631379325 wait complete. 2024-02-19T22:05:39.012 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:39.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:39 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:38.998986+0000 mon.smithi143 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.173:0/1631379325' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:40.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:40.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:39.340988+0000 mgr.smithi143.bissbc (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:42.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.835+0000 7fdb455ac700 1 -- 172.21.15.173:0/3648934595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb400f6510 msgr2=0x7fdb400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:42.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.835+0000 7fdb455ac700 1 --2- 172.21.15.173:0/3648934595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb400f6510 0x7fdb400f6930 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fdb34009fb0 tx=0x7fdb34009b30).stop 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.835+0000 7fdb455ac700 1 -- 172.21.15.173:0/3648934595 shutdown_connections 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.835+0000 7fdb455ac700 1 --2- 172.21.15.173:0/3648934595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb400f6510 0x7fdb400f6930 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.835+0000 7fdb455ac700 1 -- 172.21.15.173:0/3648934595 >> 172.21.15.173:0/3648934595 conn(0x7fdb400f1c70 msgr2=0x7fdb400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.835+0000 7fdb455ac700 1 -- 172.21.15.173:0/3648934595 shutdown_connections 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.835+0000 7fdb455ac700 1 -- 172.21.15.173:0/3648934595 wait complete. 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.836+0000 7fdb455ac700 1 Processor -- start 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.836+0000 7fdb455ac700 1 -- start start 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.836+0000 7fdb455ac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb400f6510 0x7fdb4006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.836+0000 7fdb455ac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb40069e20 con 0x7fdb400f6510 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.836+0000 7fdb3effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb400f6510 0x7fdb4006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.836+0000 7fdb3effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb400f6510 0x7fdb4006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:42892/0 (socket says 172.21.15.173:42892) 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.836+0000 7fdb3effd700 1 -- 172.21.15.173:0/2649764893 learned_addr learned my addr 172.21.15.173:0/2649764893 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.836+0000 7fdb3effd700 1 -- 172.21.15.173:0/2649764893 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb34005040 con 0x7fdb400f6510 2024-02-19T22:05:42.837 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.836+0000 7fdb3effd700 1 --2- 172.21.15.173:0/2649764893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb400f6510 0x7fdb4006b9a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fdb34005010 tx=0x7fdb3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:42.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.837+0000 7fdb277fe700 1 -- 172.21.15.173:0/2649764893 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb34009320 con 0x7fdb400f6510 2024-02-19T22:05:42.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.837+0000 7fdb277fe700 1 -- 172.21.15.173:0/2649764893 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb34013380 con 0x7fdb400f6510 2024-02-19T22:05:42.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.837+0000 7fdb455ac700 1 -- 172.21.15.173:0/2649764893 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb40069fc0 con 0x7fdb400f6510 2024-02-19T22:05:42.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.837+0000 7fdb277fe700 1 -- 172.21.15.173:0/2649764893 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb34014e50 con 0x7fdb400f6510 2024-02-19T22:05:42.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.837+0000 7fdb455ac700 1 -- 172.21.15.173:0/2649764893 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4006a450 con 0x7fdb400f6510 2024-02-19T22:05:42.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.837+0000 7fdb277fe700 1 -- 172.21.15.173:0/2649764893 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb34014420 con 0x7fdb400f6510 2024-02-19T22:05:42.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.838+0000 7fdb455ac700 1 -- 172.21.15.173:0/2649764893 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb40059100 con 0x7fdb400f6510 2024-02-19T22:05:42.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.838+0000 7fdb277fe700 1 --2- 172.21.15.173:0/2649764893 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdb2802ff10 0x7fdb280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:42.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.838+0000 7fdb277fe700 1 -- 172.21.15.173:0/2649764893 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb34040e30 con 0x7fdb400f6510 2024-02-19T22:05:42.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.838+0000 7fdb3e7fc700 1 --2- 172.21.15.173:0/2649764893 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdb2802ff10 0x7fdb280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:42.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.838+0000 7fdb3e7fc700 1 --2- 172.21.15.173:0/2649764893 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdb2802ff10 0x7fdb280323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fdb300096a0 tx=0x7fdb30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:42.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:42.843+0000 7fdb277fe700 1 -- 172.21.15.173:0/2649764893 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb340095f0 con 0x7fdb400f6510 2024-02-19T22:05:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:41.341270+0000 mgr.smithi143.bissbc (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:43.053 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.053+0000 7fdb455ac700 1 -- 172.21.15.173:0/2649764893 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb400fc0a0 con 0x7fdb400f6510 2024-02-19T22:05:43.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.054+0000 7fdb277fe700 1 -- 172.21.15.173:0/2649764893 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdb3401db70 con 0x7fdb400f6510 2024-02-19T22:05:43.054 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:43.054 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:43.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.056+0000 7fdb455ac700 1 -- 172.21.15.173:0/2649764893 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdb2802ff10 msgr2=0x7fdb280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:43.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.056+0000 7fdb455ac700 1 --2- 172.21.15.173:0/2649764893 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdb2802ff10 0x7fdb280323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fdb300096a0 tx=0x7fdb30006b40).stop 2024-02-19T22:05:43.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.056+0000 7fdb455ac700 1 -- 172.21.15.173:0/2649764893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb400f6510 msgr2=0x7fdb4006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:43.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.056+0000 7fdb455ac700 1 --2- 172.21.15.173:0/2649764893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb400f6510 0x7fdb4006b9a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fdb34005010 tx=0x7fdb3400b040).stop 2024-02-19T22:05:43.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.056+0000 7fdb455ac700 1 -- 172.21.15.173:0/2649764893 shutdown_connections 2024-02-19T22:05:43.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.056+0000 7fdb455ac700 1 --2- 172.21.15.173:0/2649764893 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fdb2802ff10 0x7fdb280323d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:43.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.056+0000 7fdb455ac700 1 --2- 172.21.15.173:0/2649764893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb400f6510 0x7fdb4006b9a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:43.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.056+0000 7fdb455ac700 1 -- 172.21.15.173:0/2649764893 >> 172.21.15.173:0/2649764893 conn(0x7fdb400f1c70 msgr2=0x7fdb400f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:43.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.056+0000 7fdb455ac700 1 -- 172.21.15.173:0/2649764893 shutdown_connections 2024-02-19T22:05:43.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:43.056+0000 7fdb455ac700 1 -- 172.21.15.173:0/2649764893 wait complete. 2024-02-19T22:05:43.057 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:43.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:43 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:43.055023+0000 mon.smithi143 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.173:0/2649764893' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:44.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:44.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:43.341563+0000 mgr.smithi143.bissbc (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:45.341907+0000 mgr.smithi143.bissbc (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:46.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.907+0000 7fb7f194a700 1 -- 172.21.15.173:0/4124251853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7ec0f5330 msgr2=0x7fb7ec0f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:46.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.907+0000 7fb7f194a700 1 --2- 172.21.15.173:0/4124251853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7ec0f5330 0x7fb7ec0f5750 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fb7d4009fb0 tx=0x7fb7d4009b30).stop 2024-02-19T22:05:46.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.907+0000 7fb7f194a700 1 -- 172.21.15.173:0/4124251853 shutdown_connections 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.907+0000 7fb7f194a700 1 --2- 172.21.15.173:0/4124251853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7ec0f5330 0x7fb7ec0f5750 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.907+0000 7fb7f194a700 1 -- 172.21.15.173:0/4124251853 >> 172.21.15.173:0/4124251853 conn(0x7fb7ec0f0a90 msgr2=0x7fb7ec0f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.907+0000 7fb7f194a700 1 -- 172.21.15.173:0/4124251853 shutdown_connections 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.907+0000 7fb7f194a700 1 -- 172.21.15.173:0/4124251853 wait complete. 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.908+0000 7fb7f194a700 1 Processor -- start 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.908+0000 7fb7f194a700 1 -- start start 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.908+0000 7fb7f194a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7ec0f5330 0x7fb7ec0fd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.908+0000 7fb7f194a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0fb900 con 0x7fb7ec0f5330 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.908+0000 7fb7eaffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7ec0f5330 0x7fb7ec0fd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.908+0000 7fb7eaffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7ec0f5330 0x7fb7ec0fd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:42910/0 (socket says 172.21.15.173:42910) 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.908+0000 7fb7eaffd700 1 -- 172.21.15.173:0/4006019042 learned_addr learned my addr 172.21.15.173:0/4006019042 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.909+0000 7fb7eaffd700 1 -- 172.21.15.173:0/4006019042 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d4005040 con 0x7fb7ec0f5330 2024-02-19T22:05:46.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.909+0000 7fb7eaffd700 1 --2- 172.21.15.173:0/4006019042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7ec0f5330 0x7fb7ec0fd480 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fb7d4005010 tx=0x7fb7d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.909+0000 7fb7e37fe700 1 -- 172.21.15.173:0/4006019042 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7d4009320 con 0x7fb7ec0f5330 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.909+0000 7fb7f194a700 1 -- 172.21.15.173:0/4006019042 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec0fbaa0 con 0x7fb7ec0f5330 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.909+0000 7fb7e37fe700 1 -- 172.21.15.173:0/4006019042 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7d4013380 con 0x7fb7ec0f5330 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.909+0000 7fb7e37fe700 1 -- 172.21.15.173:0/4006019042 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7d4014e50 con 0x7fb7ec0f5330 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.909+0000 7fb7f194a700 1 -- 172.21.15.173:0/4006019042 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec0fbf30 con 0x7fb7ec0f5330 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.910+0000 7fb7e37fe700 1 -- 172.21.15.173:0/4006019042 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7d40095f0 con 0x7fb7ec0f5330 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.910+0000 7fb7f194a700 1 -- 172.21.15.173:0/4006019042 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ec045ad0 con 0x7fb7ec0f5330 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.910+0000 7fb7e37fe700 1 --2- 172.21.15.173:0/4006019042 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb7d802ff10 0x7fb7d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.910+0000 7fb7e37fe700 1 -- 172.21.15.173:0/4006019042 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb7d4040e90 con 0x7fb7ec0f5330 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.910+0000 7fb7ea7fc700 1 --2- 172.21.15.173:0/4006019042 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb7d802ff10 0x7fb7d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.911+0000 7fb7ea7fc700 1 --2- 172.21.15.173:0/4006019042 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb7d802ff10 0x7fb7d80323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb7dc0096a0 tx=0x7fb7dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:46.926 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:46.915+0000 7fb7e37fe700 1 -- 172.21.15.173:0/4006019042 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7d4014420 con 0x7fb7ec0f5330 2024-02-19T22:05:47.120 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.120+0000 7fb7f194a700 1 -- 172.21.15.173:0/4006019042 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7ec05f270 con 0x7fb7ec0f5330 2024-02-19T22:05:47.120 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.121+0000 7fb7e37fe700 1 -- 172.21.15.173:0/4006019042 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb7d4010230 con 0x7fb7ec0f5330 2024-02-19T22:05:47.121 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:47.121 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:47.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.122+0000 7fb7f194a700 1 -- 172.21.15.173:0/4006019042 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb7d802ff10 msgr2=0x7fb7d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:47.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.122+0000 7fb7f194a700 1 --2- 172.21.15.173:0/4006019042 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb7d802ff10 0x7fb7d80323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb7dc0096a0 tx=0x7fb7dc006b40).stop 2024-02-19T22:05:47.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.122+0000 7fb7f194a700 1 -- 172.21.15.173:0/4006019042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7ec0f5330 msgr2=0x7fb7ec0fd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:47.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.122+0000 7fb7f194a700 1 --2- 172.21.15.173:0/4006019042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7ec0f5330 0x7fb7ec0fd480 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fb7d4005010 tx=0x7fb7d400b040).stop 2024-02-19T22:05:47.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.123+0000 7fb7f194a700 1 -- 172.21.15.173:0/4006019042 shutdown_connections 2024-02-19T22:05:47.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.123+0000 7fb7f194a700 1 --2- 172.21.15.173:0/4006019042 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb7d802ff10 0x7fb7d80323d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:47.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.123+0000 7fb7f194a700 1 --2- 172.21.15.173:0/4006019042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7ec0f5330 0x7fb7ec0fd480 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:47.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.123+0000 7fb7f194a700 1 -- 172.21.15.173:0/4006019042 >> 172.21.15.173:0/4006019042 conn(0x7fb7ec0f0a90 msgr2=0x7fb7ec05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:47.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.123+0000 7fb7f194a700 1 -- 172.21.15.173:0/4006019042 shutdown_connections 2024-02-19T22:05:47.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:47.123+0000 7fb7f194a700 1 -- 172.21.15.173:0/4006019042 wait complete. 2024-02-19T22:05:47.123 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:47.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:47 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:47.122075+0000 mon.smithi143 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.173:0/4006019042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:48.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:48.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:47.342222+0000 mgr.smithi143.bissbc (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:49.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:49 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:49.342532+0000 mgr.smithi143.bissbc (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:50.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.827+0000 7f0883c23700 1 -- 172.21.15.173:0/2660017760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c0f5410 msgr2=0x7f087c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:50.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.827+0000 7f0883c23700 1 --2- 172.21.15.173:0/2660017760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c0f5410 0x7f087c0f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f086c009fb0 tx=0x7f086c009b30).stop 2024-02-19T22:05:50.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.827+0000 7f0883c23700 1 -- 172.21.15.173:0/2660017760 shutdown_connections 2024-02-19T22:05:50.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.827+0000 7f0883c23700 1 --2- 172.21.15.173:0/2660017760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c0f5410 0x7f087c0f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.827+0000 7f0883c23700 1 -- 172.21.15.173:0/2660017760 >> 172.21.15.173:0/2660017760 conn(0x7f087c0f0b70 msgr2=0x7f087c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.827+0000 7f0883c23700 1 -- 172.21.15.173:0/2660017760 shutdown_connections 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.827+0000 7f0883c23700 1 -- 172.21.15.173:0/2660017760 wait complete. 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.828+0000 7f0883c23700 1 Processor -- start 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.828+0000 7f0883c23700 1 -- start start 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.828+0000 7f0883c23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c0f5410 0x7f087c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.828+0000 7f0883c23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c105bd0 con 0x7f087c0f5410 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.828+0000 7f08819bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c0f5410 0x7f087c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.828+0000 7f08819bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c0f5410 0x7f087c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:42918/0 (socket says 172.21.15.173:42918) 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.828+0000 7f08819bf700 1 -- 172.21.15.173:0/596253187 learned_addr learned my addr 172.21.15.173:0/596253187 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.829+0000 7f08819bf700 1 -- 172.21.15.173:0/596253187 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c005040 con 0x7f087c0f5410 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.829+0000 7f08819bf700 1 --2- 172.21.15.173:0/596253187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c0f5410 0x7f087c1056e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f086c005010 tx=0x7f086c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:50.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.829+0000 7f08727fc700 1 -- 172.21.15.173:0/596253187 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f086c009320 con 0x7f087c0f5410 2024-02-19T22:05:50.831 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.829+0000 7f0883c23700 1 -- 172.21.15.173:0/596253187 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c105dd0 con 0x7f087c0f5410 2024-02-19T22:05:50.831 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.829+0000 7f08727fc700 1 -- 172.21.15.173:0/596253187 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f086c013380 con 0x7f087c0f5410 2024-02-19T22:05:50.831 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.829+0000 7f08727fc700 1 -- 172.21.15.173:0/596253187 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f086c014e50 con 0x7f087c0f5410 2024-02-19T22:05:50.831 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.829+0000 7f0883c23700 1 -- 172.21.15.173:0/596253187 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c106260 con 0x7f087c0f5410 2024-02-19T22:05:50.831 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.830+0000 7f08727fc700 1 -- 172.21.15.173:0/596253187 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f086c0263a0 con 0x7f087c0f5410 2024-02-19T22:05:50.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.830+0000 7f08727fc700 1 --2- 172.21.15.173:0/596253187 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f086802ff10 0x7f08680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:50.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.830+0000 7f0883c23700 1 -- 172.21.15.173:0/596253187 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c045ad0 con 0x7f087c0f5410 2024-02-19T22:05:50.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.830+0000 7f08727fc700 1 -- 172.21.15.173:0/596253187 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f086c041140 con 0x7f087c0f5410 2024-02-19T22:05:50.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.830+0000 7f08811be700 1 --2- 172.21.15.173:0/596253187 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f086802ff10 0x7f08680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:50.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.831+0000 7f08811be700 1 --2- 172.21.15.173:0/596253187 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f086802ff10 0x7f08680323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f08780096a0 tx=0x7f0878006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:50.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:50.835+0000 7f08727fc700 1 -- 172.21.15.173:0/596253187 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f086c01ccf0 con 0x7f087c0f5410 2024-02-19T22:05:51.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.038+0000 7f0883c23700 1 -- 172.21.15.173:0/596253187 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f087c107110 con 0x7f087c0f5410 2024-02-19T22:05:51.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.039+0000 7f08727fc700 1 -- 172.21.15.173:0/596253187 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f086c012330 con 0x7f087c0f5410 2024-02-19T22:05:51.039 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:51.039 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:51.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.040+0000 7f0883c23700 1 -- 172.21.15.173:0/596253187 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f086802ff10 msgr2=0x7f08680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:51.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.040+0000 7f0883c23700 1 --2- 172.21.15.173:0/596253187 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f086802ff10 0x7f08680323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f08780096a0 tx=0x7f0878006b40).stop 2024-02-19T22:05:51.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.040+0000 7f0883c23700 1 -- 172.21.15.173:0/596253187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c0f5410 msgr2=0x7f087c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:51.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.040+0000 7f0883c23700 1 --2- 172.21.15.173:0/596253187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c0f5410 0x7f087c1056e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f086c005010 tx=0x7f086c00b040).stop 2024-02-19T22:05:51.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.040+0000 7f0883c23700 1 -- 172.21.15.173:0/596253187 shutdown_connections 2024-02-19T22:05:51.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.040+0000 7f0883c23700 1 --2- 172.21.15.173:0/596253187 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f086802ff10 0x7f08680323d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:51.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.040+0000 7f0883c23700 1 --2- 172.21.15.173:0/596253187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c0f5410 0x7f087c1056e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:51.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.040+0000 7f0883c23700 1 -- 172.21.15.173:0/596253187 >> 172.21.15.173:0/596253187 conn(0x7f087c0f0b70 msgr2=0x7f087c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:51.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.040+0000 7f0883c23700 1 -- 172.21.15.173:0/596253187 shutdown_connections 2024-02-19T22:05:51.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:51.041+0000 7f0883c23700 1 -- 172.21.15.173:0/596253187 wait complete. 2024-02-19T22:05:51.041 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:51.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:51 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:51.040120+0000 mon.smithi143 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.173:0/596253187' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:52.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:51.342943+0000 mgr.smithi143.bissbc (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:52.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:52.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:54.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.742+0000 7f1ca1037700 1 -- 172.21.15.173:0/3131584767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c9c0f62a0 msgr2=0x7f1c9c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.742+0000 7f1ca1037700 1 --2- 172.21.15.173:0/3131584767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c9c0f62a0 0x7f1c9c0f66c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f1c84009fb0 tx=0x7f1c84009b30).stop 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.742+0000 7f1ca1037700 1 -- 172.21.15.173:0/3131584767 shutdown_connections 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.742+0000 7f1ca1037700 1 --2- 172.21.15.173:0/3131584767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c9c0f62a0 0x7f1c9c0f66c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.742+0000 7f1ca1037700 1 -- 172.21.15.173:0/3131584767 >> 172.21.15.173:0/3131584767 conn(0x7f1c9c0f1aa0 msgr2=0x7f1c9c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.742+0000 7f1ca1037700 1 -- 172.21.15.173:0/3131584767 shutdown_connections 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.742+0000 7f1ca1037700 1 -- 172.21.15.173:0/3131584767 wait complete. 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.742+0000 7f1ca1037700 1 Processor -- start 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.743+0000 7f1ca1037700 1 -- start start 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.743+0000 7f1ca1037700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c9c1066c0 0x7f1c9c106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.743+0000 7f1ca1037700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c106fd0 con 0x7f1c9c1066c0 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.743+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c9c1066c0 0x7f1c9c106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.743+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c9c1066c0 0x7f1c9c106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:39868/0 (socket says 172.21.15.173:39868) 2024-02-19T22:05:54.745 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.743+0000 7f1c9bfff700 1 -- 172.21.15.173:0/679595099 learned_addr learned my addr 172.21.15.173:0/679595099 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.743+0000 7f1c9bfff700 1 -- 172.21.15.173:0/679595099 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c84005040 con 0x7f1c9c1066c0 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.743+0000 7f1c9bfff700 1 --2- 172.21.15.173:0/679595099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c9c1066c0 0x7f1c9c106ae0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f1c84009f80 tx=0x7f1c84009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.744+0000 7f1c98ff9700 1 -- 172.21.15.173:0/679595099 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c84013380 con 0x7f1c9c1066c0 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.744+0000 7f1ca1037700 1 -- 172.21.15.173:0/679595099 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c9c1071d0 con 0x7f1c9c1066c0 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.744+0000 7f1c98ff9700 1 -- 172.21.15.173:0/679595099 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c84013790 con 0x7f1c9c1066c0 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.744+0000 7f1c98ff9700 1 -- 172.21.15.173:0/679595099 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c84014420 con 0x7f1c9c1066c0 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.744+0000 7f1ca1037700 1 -- 172.21.15.173:0/679595099 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9c107d20 con 0x7f1c9c1066c0 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.744+0000 7f1c98ff9700 1 -- 172.21.15.173:0/679595099 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c84025060 con 0x7f1c9c1066c0 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.744+0000 7f1ca1037700 1 -- 172.21.15.173:0/679595099 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9c059100 con 0x7f1c9c1066c0 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.745+0000 7f1c98ff9700 1 --2- 172.21.15.173:0/679595099 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c8802ff60 0x7f1c88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.745+0000 7f1c98ff9700 1 -- 172.21.15.173:0/679595099 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c84040e10 con 0x7f1c9c1066c0 2024-02-19T22:05:54.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.745+0000 7f1c9b7fe700 1 --2- 172.21.15.173:0/679595099 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c8802ff60 0x7f1c88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:54.747 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.745+0000 7f1c9b7fe700 1 --2- 172.21.15.173:0/679595099 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c8802ff60 0x7f1c88032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f1c8c0096a0 tx=0x7f1c8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:54.749 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.750+0000 7f1c98ff9700 1 -- 172.21.15.173:0/679595099 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c8401cb60 con 0x7f1c9c1066c0 2024-02-19T22:05:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:53.343321+0000 mgr.smithi143.bissbc (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:54.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.957+0000 7f1ca1037700 1 -- 172.21.15.173:0/679595099 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c9c045ad0 con 0x7f1c9c1066c0 2024-02-19T22:05:54.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.957+0000 7f1c98ff9700 1 -- 172.21.15.173:0/679595099 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1c84042030 con 0x7f1c9c1066c0 2024-02-19T22:05:54.957 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:54.957 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:54.958 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.959+0000 7f1ca1037700 1 -- 172.21.15.173:0/679595099 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c8802ff60 msgr2=0x7f1c88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:54.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.959+0000 7f1ca1037700 1 --2- 172.21.15.173:0/679595099 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c8802ff60 0x7f1c88032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f1c8c0096a0 tx=0x7f1c8c006b40).stop 2024-02-19T22:05:54.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.959+0000 7f1ca1037700 1 -- 172.21.15.173:0/679595099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c9c1066c0 msgr2=0x7f1c9c106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:54.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.959+0000 7f1ca1037700 1 --2- 172.21.15.173:0/679595099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c9c1066c0 0x7f1c9c106ae0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f1c84009f80 tx=0x7f1c84009b30).stop 2024-02-19T22:05:54.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.959+0000 7f1ca1037700 1 -- 172.21.15.173:0/679595099 shutdown_connections 2024-02-19T22:05:54.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.959+0000 7f1ca1037700 1 --2- 172.21.15.173:0/679595099 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c8802ff60 0x7f1c88032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:54.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.959+0000 7f1ca1037700 1 --2- 172.21.15.173:0/679595099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c9c1066c0 0x7f1c9c106ae0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:54.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.959+0000 7f1ca1037700 1 -- 172.21.15.173:0/679595099 >> 172.21.15.173:0/679595099 conn(0x7f1c9c0f1aa0 msgr2=0x7f1c9c0f2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:54.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.959+0000 7f1ca1037700 1 -- 172.21.15.173:0/679595099 shutdown_connections 2024-02-19T22:05:54.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:54.959+0000 7f1ca1037700 1 -- 172.21.15.173:0/679595099 wait complete. 2024-02-19T22:05:54.959 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:55.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:55 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:54.958479+0000 mon.smithi143 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.173:0/679595099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:56.684 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:56.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:55.343640+0000 mgr.smithi143.bissbc (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:58.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.732+0000 7fbb449a0700 1 -- 172.21.15.173:0/919788517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb400f51a0 msgr2=0x7fbb400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:58.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.732+0000 7fbb449a0700 1 --2- 172.21.15.173:0/919788517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb400f51a0 0x7fbb400f55c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fbb30009fb0 tx=0x7fbb30009b30).stop 2024-02-19T22:05:58.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.733+0000 7fbb449a0700 1 -- 172.21.15.173:0/919788517 shutdown_connections 2024-02-19T22:05:58.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.733+0000 7fbb449a0700 1 --2- 172.21.15.173:0/919788517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb400f51a0 0x7fbb400f55c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:58.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.733+0000 7fbb449a0700 1 -- 172.21.15.173:0/919788517 >> 172.21.15.173:0/919788517 conn(0x7fbb400f09a0 msgr2=0x7fbb400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:58.736 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.733+0000 7fbb449a0700 1 -- 172.21.15.173:0/919788517 shutdown_connections 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.733+0000 7fbb449a0700 1 -- 172.21.15.173:0/919788517 wait complete. 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.733+0000 7fbb449a0700 1 Processor -- start 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.733+0000 7fbb449a0700 1 -- start start 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.733+0000 7fbb449a0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb400f51a0 0x7fbb40109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.733+0000 7fbb449a0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb40060280 con 0x7fbb400f51a0 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.734+0000 7fbb3f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb400f51a0 0x7fbb40109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.734+0000 7fbb3f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb400f51a0 0x7fbb40109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:39886/0 (socket says 172.21.15.173:39886) 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.734+0000 7fbb3f7fe700 1 -- 172.21.15.173:0/1967540866 learned_addr learned my addr 172.21.15.173:0/1967540866 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.734+0000 7fbb3f7fe700 1 -- 172.21.15.173:0/1967540866 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb30005040 con 0x7fbb400f51a0 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.734+0000 7fbb3f7fe700 1 --2- 172.21.15.173:0/1967540866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb400f51a0 0x7fbb40109290 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fbb30009830 tx=0x7fbb3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.734+0000 7fbb27fff700 1 -- 172.21.15.173:0/1967540866 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb30009320 con 0x7fbb400f51a0 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.734+0000 7fbb27fff700 1 -- 172.21.15.173:0/1967540866 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb30013380 con 0x7fbb400f51a0 2024-02-19T22:05:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.734+0000 7fbb449a0700 1 -- 172.21.15.173:0/1967540866 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb40060480 con 0x7fbb400f51a0 2024-02-19T22:05:58.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.734+0000 7fbb27fff700 1 -- 172.21.15.173:0/1967540866 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb30014420 con 0x7fbb400f51a0 2024-02-19T22:05:58.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.735+0000 7fbb449a0700 1 -- 172.21.15.173:0/1967540866 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb401099c0 con 0x7fbb400f51a0 2024-02-19T22:05:58.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.735+0000 7fbb27fff700 1 -- 172.21.15.173:0/1967540866 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb30014580 con 0x7fbb400f51a0 2024-02-19T22:05:58.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.735+0000 7fbb449a0700 1 -- 172.21.15.173:0/1967540866 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb40045ad0 con 0x7fbb400f51a0 2024-02-19T22:05:58.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.735+0000 7fbb27fff700 1 --2- 172.21.15.173:0/1967540866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbb2802ff60 0x7fbb28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:58.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.735+0000 7fbb27fff700 1 -- 172.21.15.173:0/1967540866 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb30041120 con 0x7fbb400f51a0 2024-02-19T22:05:58.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.736+0000 7fbb3effd700 1 --2- 172.21.15.173:0/1967540866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbb2802ff60 0x7fbb28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:58.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.736+0000 7fbb3effd700 1 --2- 172.21.15.173:0/1967540866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbb2802ff60 0x7fbb28032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fbb340096a0 tx=0x7fbb34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:58.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.740+0000 7fbb27fff700 1 -- 172.21.15.173:0/1967540866 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb30014db0 con 0x7fbb400f51a0 2024-02-19T22:05:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:57.343942+0000 mgr.smithi143.bissbc (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:58.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.949+0000 7fbb449a0700 1 -- 172.21.15.173:0/1967540866 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb400fad30 con 0x7fbb400f51a0 2024-02-19T22:05:58.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.949+0000 7fbb27fff700 1 -- 172.21.15.173:0/1967540866 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbb30004490 con 0x7fbb400f51a0 2024-02-19T22:05:58.949 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:05:58.950 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:58.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.951+0000 7fbb449a0700 1 -- 172.21.15.173:0/1967540866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbb2802ff60 msgr2=0x7fbb28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:58.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.951+0000 7fbb449a0700 1 --2- 172.21.15.173:0/1967540866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbb2802ff60 0x7fbb28032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fbb340096a0 tx=0x7fbb34006b40).stop 2024-02-19T22:05:58.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.951+0000 7fbb449a0700 1 -- 172.21.15.173:0/1967540866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb400f51a0 msgr2=0x7fbb40109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:58.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.951+0000 7fbb449a0700 1 --2- 172.21.15.173:0/1967540866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb400f51a0 0x7fbb40109290 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fbb30009830 tx=0x7fbb3000b040).stop 2024-02-19T22:05:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.951+0000 7fbb449a0700 1 -- 172.21.15.173:0/1967540866 shutdown_connections 2024-02-19T22:05:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.951+0000 7fbb449a0700 1 --2- 172.21.15.173:0/1967540866 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fbb2802ff60 0x7fbb28032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.951+0000 7fbb449a0700 1 --2- 172.21.15.173:0/1967540866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb400f51a0 0x7fbb40109290 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.951+0000 7fbb449a0700 1 -- 172.21.15.173:0/1967540866 >> 172.21.15.173:0/1967540866 conn(0x7fbb400f09a0 msgr2=0x7fbb400f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.951+0000 7fbb449a0700 1 -- 172.21.15.173:0/1967540866 shutdown_connections 2024-02-19T22:05:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:05:58.951+0000 7fbb449a0700 1 -- 172.21.15.173:0/1967540866 wait complete. 2024-02-19T22:05:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:05:59.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:05:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:05:58.950645+0000 mon.smithi143 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.173:0/1967540866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:00.716 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:00.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:00.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:00 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:05:59.344196+0000 mgr.smithi143.bissbc (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:01.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:01 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:01.344486+0000 mgr.smithi143.bissbc (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:02.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.769+0000 7f3ee4b56700 1 -- 172.21.15.173:0/631792109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ee00f4b80 msgr2=0x7f3ee00f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:02.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.769+0000 7f3ee4b56700 1 --2- 172.21.15.173:0/631792109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ee00f4b80 0x7f3ee00f4fa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f3ec8004770 tx=0x7f3ec800ac40).stop 2024-02-19T22:06:02.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.769+0000 7f3ee4b56700 1 -- 172.21.15.173:0/631792109 shutdown_connections 2024-02-19T22:06:02.771 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.769+0000 7f3ee4b56700 1 --2- 172.21.15.173:0/631792109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ee00f4b80 0x7f3ee00f4fa0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.769+0000 7f3ee4b56700 1 -- 172.21.15.173:0/631792109 >> 172.21.15.173:0/631792109 conn(0x7f3ee00f02e0 msgr2=0x7f3ee00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.769+0000 7f3ee4b56700 1 -- 172.21.15.173:0/631792109 shutdown_connections 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.769+0000 7f3ee4b56700 1 -- 172.21.15.173:0/631792109 wait complete. 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.769+0000 7f3ee4b56700 1 Processor -- start 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.769+0000 7f3ee4b56700 1 -- start start 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.769+0000 7f3ee4b56700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ee00f4b80 0x7f3ee0106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.769+0000 7f3ee4b56700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee0106c80 con 0x7f3ee00f4b80 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.770+0000 7f3ede59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ee00f4b80 0x7f3ee0106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.770+0000 7f3ede59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ee00f4b80 0x7f3ee0106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:58582/0 (socket says 172.21.15.173:58582) 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.770+0000 7f3ede59c700 1 -- 172.21.15.173:0/1700779771 learned_addr learned my addr 172.21.15.173:0/1700779771 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.770+0000 7f3ede59c700 1 -- 172.21.15.173:0/1700779771 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec8005040 con 0x7f3ee00f4b80 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.770+0000 7f3ede59c700 1 --2- 172.21.15.173:0/1700779771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ee00f4b80 0x7f3ee0106790 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f3ec800aaa0 tx=0x7f3ec800ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:02.772 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.770+0000 7f3ed6ffd700 1 -- 172.21.15.173:0/1700779771 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ec8019070 con 0x7f3ee00f4b80 2024-02-19T22:06:02.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.770+0000 7f3ee4b56700 1 -- 172.21.15.173:0/1700779771 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee0106e80 con 0x7f3ee00f4b80 2024-02-19T22:06:02.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.770+0000 7f3ed6ffd700 1 -- 172.21.15.173:0/1700779771 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ec80148d0 con 0x7f3ee00f4b80 2024-02-19T22:06:02.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.770+0000 7f3ed6ffd700 1 -- 172.21.15.173:0/1700779771 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ec8008480 con 0x7f3ee00f4b80 2024-02-19T22:06:02.773 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.770+0000 7f3ee4b56700 1 -- 172.21.15.173:0/1700779771 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee0107310 con 0x7f3ee00f4b80 2024-02-19T22:06:02.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.771+0000 7f3ed6ffd700 1 -- 172.21.15.173:0/1700779771 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ec8025070 con 0x7f3ee00f4b80 2024-02-19T22:06:02.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.771+0000 7f3ee4b56700 1 -- 172.21.15.173:0/1700779771 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee0045ad0 con 0x7f3ee00f4b80 2024-02-19T22:06:02.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.771+0000 7f3ed6ffd700 1 --2- 172.21.15.173:0/1700779771 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ecc02ff60 0x7f3ecc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:02.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.771+0000 7f3ed6ffd700 1 -- 172.21.15.173:0/1700779771 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ec80409d0 con 0x7f3ee00f4b80 2024-02-19T22:06:02.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.771+0000 7f3eddd9b700 1 --2- 172.21.15.173:0/1700779771 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ecc02ff60 0x7f3ecc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:02.774 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.772+0000 7f3eddd9b700 1 --2- 172.21.15.173:0/1700779771 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ecc02ff60 0x7f3ecc032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f3ed00096a0 tx=0x7f3ed0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:02.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.777+0000 7f3ed6ffd700 1 -- 172.21.15.173:0/1700779771 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ec8025770 con 0x7f3ee00f4b80 2024-02-19T22:06:02.985 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.984+0000 7f3ee4b56700 1 -- 172.21.15.173:0/1700779771 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ee01081c0 con 0x7f3ee00f4b80 2024-02-19T22:06:02.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.986+0000 7f3ed6ffd700 1 -- 172.21.15.173:0/1700779771 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ec8025770 con 0x7f3ee00f4b80 2024-02-19T22:06:02.986 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:02.987 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:02.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.987+0000 7f3ee4b56700 1 -- 172.21.15.173:0/1700779771 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ecc02ff60 msgr2=0x7f3ecc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:02.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.987+0000 7f3ee4b56700 1 --2- 172.21.15.173:0/1700779771 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ecc02ff60 0x7f3ecc032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f3ed00096a0 tx=0x7f3ed0006b40).stop 2024-02-19T22:06:02.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.987+0000 7f3ee4b56700 1 -- 172.21.15.173:0/1700779771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ee00f4b80 msgr2=0x7f3ee0106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:02.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.987+0000 7f3ee4b56700 1 --2- 172.21.15.173:0/1700779771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ee00f4b80 0x7f3ee0106790 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f3ec800aaa0 tx=0x7f3ec800ac40).stop 2024-02-19T22:06:02.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.987+0000 7f3ee4b56700 1 -- 172.21.15.173:0/1700779771 shutdown_connections 2024-02-19T22:06:02.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.987+0000 7f3ee4b56700 1 --2- 172.21.15.173:0/1700779771 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f3ecc02ff60 0x7f3ecc032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:02.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.987+0000 7f3ee4b56700 1 --2- 172.21.15.173:0/1700779771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ee00f4b80 0x7f3ee0106790 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:02.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.987+0000 7f3ee4b56700 1 -- 172.21.15.173:0/1700779771 >> 172.21.15.173:0/1700779771 conn(0x7f3ee00f02e0 msgr2=0x7f3ee00f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:02.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.987+0000 7f3ee4b56700 1 -- 172.21.15.173:0/1700779771 shutdown_connections 2024-02-19T22:06:02.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:02.987+0000 7f3ee4b56700 1 -- 172.21.15.173:0/1700779771 wait complete. 2024-02-19T22:06:02.990 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:03.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:03 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:02.986911+0000 mon.smithi143 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.173:0/1700779771' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:04.748 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:04.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:04.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:04 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:03.344818+0000 mgr.smithi143.bissbc (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:06.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:06 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:05.345112+0000 mgr.smithi143.bissbc (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:06.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.951+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/3221312115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e780f6770 msgr2=0x7f6e780f6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:06.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.951+0000 7f6e7ebb2700 1 --2- 172.21.15.173:0/3221312115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e780f6770 0x7f6e780f6b90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f6e74009fb0 tx=0x7f6e74009b30).stop 2024-02-19T22:06:06.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.951+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/3221312115 shutdown_connections 2024-02-19T22:06:06.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.951+0000 7f6e7ebb2700 1 --2- 172.21.15.173:0/3221312115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e780f6770 0x7f6e780f6b90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:06.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.951+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/3221312115 >> 172.21.15.173:0/3221312115 conn(0x7f6e780f0120 msgr2=0x7f6e780f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:06.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.951+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/3221312115 shutdown_connections 2024-02-19T22:06:06.952 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.951+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/3221312115 wait complete. 2024-02-19T22:06:06.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.952+0000 7f6e7ebb2700 1 Processor -- start 2024-02-19T22:06:06.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.952+0000 7f6e7ebb2700 1 -- start start 2024-02-19T22:06:06.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.952+0000 7f6e7ebb2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e780f6770 0x7f6e781092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:06.955 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.952+0000 7f6e7ebb2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780f50b0 con 0x7f6e780f6770 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.952+0000 7f6e7dbb0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e780f6770 0x7f6e781092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.952+0000 7f6e7dbb0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e780f6770 0x7f6e781092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:58600/0 (socket says 172.21.15.173:58600) 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.952+0000 7f6e7dbb0700 1 -- 172.21.15.173:0/1074807729 learned_addr learned my addr 172.21.15.173:0/1074807729 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.953+0000 7f6e7dbb0700 1 -- 172.21.15.173:0/1074807729 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74005040 con 0x7f6e780f6770 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.953+0000 7f6e7dbb0700 1 --2- 172.21.15.173:0/1074807729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e780f6770 0x7f6e781092d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f6e74005010 tx=0x7f6e7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.953+0000 7f6e6e7fc700 1 -- 172.21.15.173:0/1074807729 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e74009320 con 0x7f6e780f6770 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.953+0000 7f6e6e7fc700 1 -- 172.21.15.173:0/1074807729 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e74013380 con 0x7f6e780f6770 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.953+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/1074807729 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e780f52b0 con 0x7f6e780f6770 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.953+0000 7f6e6e7fc700 1 -- 172.21.15.173:0/1074807729 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e74014e50 con 0x7f6e780f6770 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.953+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/1074807729 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e78109980 con 0x7f6e780f6770 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.954+0000 7f6e6e7fc700 1 -- 172.21.15.173:0/1074807729 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e740263a0 con 0x7f6e780f6770 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.954+0000 7f6e6e7fc700 1 --2- 172.21.15.173:0/1074807729 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6e6402ff10 0x7f6e640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:06.956 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.954+0000 7f6e6e7fc700 1 -- 172.21.15.173:0/1074807729 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e74040fb0 con 0x7f6e780f6770 2024-02-19T22:06:06.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.954+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/1074807729 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e78045ad0 con 0x7f6e780f6770 2024-02-19T22:06:06.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.955+0000 7f6e7d3af700 1 --2- 172.21.15.173:0/1074807729 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6e6402ff10 0x7f6e640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:06.957 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.955+0000 7f6e7d3af700 1 --2- 172.21.15.173:0/1074807729 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6e6402ff10 0x7f6e640323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6e680096a0 tx=0x7f6e68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:06.959 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:06.960+0000 7f6e6e7fc700 1 -- 172.21.15.173:0/1074807729 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e74012920 con 0x7f6e780f6770 2024-02-19T22:06:07.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.164+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/1074807729 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e780fae00 con 0x7f6e780f6770 2024-02-19T22:06:07.165 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.165+0000 7f6e6e7fc700 1 -- 172.21.15.173:0/1074807729 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6e7402b020 con 0x7f6e780f6770 2024-02-19T22:06:07.165 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:07.165 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:07.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.166+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/1074807729 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6e6402ff10 msgr2=0x7f6e640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:07.166 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.166+0000 7f6e7ebb2700 1 --2- 172.21.15.173:0/1074807729 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6e6402ff10 0x7f6e640323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6e680096a0 tx=0x7f6e68006b40).stop 2024-02-19T22:06:07.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.166+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/1074807729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e780f6770 msgr2=0x7f6e781092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:07.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.166+0000 7f6e7ebb2700 1 --2- 172.21.15.173:0/1074807729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e780f6770 0x7f6e781092d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f6e74005010 tx=0x7f6e7400b040).stop 2024-02-19T22:06:07.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.166+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/1074807729 shutdown_connections 2024-02-19T22:06:07.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.166+0000 7f6e7ebb2700 1 --2- 172.21.15.173:0/1074807729 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f6e6402ff10 0x7f6e640323d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:07.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.166+0000 7f6e7ebb2700 1 --2- 172.21.15.173:0/1074807729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e780f6770 0x7f6e781092d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:07.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.166+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/1074807729 >> 172.21.15.173:0/1074807729 conn(0x7f6e780f0120 msgr2=0x7f6e780f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:07.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.167+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/1074807729 shutdown_connections 2024-02-19T22:06:07.167 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:07.167+0000 7f6e7ebb2700 1 -- 172.21.15.173:0/1074807729 wait complete. 2024-02-19T22:06:07.168 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:07.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:07 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:07.166183+0000 mon.smithi143 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.173:0/1074807729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:08.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:08 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:07.345379+0000 mgr.smithi143.bissbc (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:08.962 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:08.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:10.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:10 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:09.345761+0000 mgr.smithi143.bissbc (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:10.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.965+0000 7ff4e6a51700 1 -- 172.21.15.173:0/680123464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e00f5410 msgr2=0x7ff4e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:10.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.965+0000 7ff4e6a51700 1 --2- 172.21.15.173:0/680123464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e00f5410 0x7ff4e00f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7ff4d4009fb0 tx=0x7ff4d4009b30).stop 2024-02-19T22:06:10.965 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.965+0000 7ff4e6a51700 1 -- 172.21.15.173:0/680123464 shutdown_connections 2024-02-19T22:06:10.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.965+0000 7ff4e6a51700 1 --2- 172.21.15.173:0/680123464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e00f5410 0x7ff4e00f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:10.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.965+0000 7ff4e6a51700 1 -- 172.21.15.173:0/680123464 >> 172.21.15.173:0/680123464 conn(0x7ff4e00f0b70 msgr2=0x7ff4e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:10.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.965+0000 7ff4e6a51700 1 -- 172.21.15.173:0/680123464 shutdown_connections 2024-02-19T22:06:10.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.965+0000 7ff4e6a51700 1 -- 172.21.15.173:0/680123464 wait complete. 2024-02-19T22:06:10.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.966+0000 7ff4e6a51700 1 Processor -- start 2024-02-19T22:06:10.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.966+0000 7ff4e6a51700 1 -- start start 2024-02-19T22:06:10.967 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.966+0000 7ff4e6a51700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e00f5410 0x7ff4e01094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.966+0000 7ff4e6a51700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e0060240 con 0x7ff4e00f5410 2024-02-19T22:06:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.967+0000 7ff4dffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e00f5410 0x7ff4e01094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.967+0000 7ff4dffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e00f5410 0x7ff4e01094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:58610/0 (socket says 172.21.15.173:58610) 2024-02-19T22:06:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.967+0000 7ff4dffff700 1 -- 172.21.15.173:0/746624630 learned_addr learned my addr 172.21.15.173:0/746624630 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.967+0000 7ff4dffff700 1 -- 172.21.15.173:0/746624630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d4005040 con 0x7ff4e00f5410 2024-02-19T22:06:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.967+0000 7ff4dffff700 1 --2- 172.21.15.173:0/746624630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e00f5410 0x7ff4e01094e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7ff4d4005010 tx=0x7ff4d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.967+0000 7ff4dcff9700 1 -- 172.21.15.173:0/746624630 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4d4009320 con 0x7ff4e00f5410 2024-02-19T22:06:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.967+0000 7ff4e6a51700 1 -- 172.21.15.173:0/746624630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e0060440 con 0x7ff4e00f5410 2024-02-19T22:06:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.967+0000 7ff4dcff9700 1 -- 172.21.15.173:0/746624630 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4d4013380 con 0x7ff4e00f5410 2024-02-19T22:06:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.967+0000 7ff4dcff9700 1 -- 172.21.15.173:0/746624630 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4d4014e50 con 0x7ff4e00f5410 2024-02-19T22:06:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.967+0000 7ff4e6a51700 1 -- 172.21.15.173:0/746624630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e0109b90 con 0x7ff4e00f5410 2024-02-19T22:06:10.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.968+0000 7ff4dcff9700 1 -- 172.21.15.173:0/746624630 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4d4014420 con 0x7ff4e00f5410 2024-02-19T22:06:10.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.968+0000 7ff4e6a51700 1 -- 172.21.15.173:0/746624630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e0045ad0 con 0x7ff4e00f5410 2024-02-19T22:06:10.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.968+0000 7ff4dcff9700 1 --2- 172.21.15.173:0/746624630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff4c802ff10 0x7ff4c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:10.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.968+0000 7ff4dcff9700 1 -- 172.21.15.173:0/746624630 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff4d4040f40 con 0x7ff4e00f5410 2024-02-19T22:06:10.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.968+0000 7ff4df7fe700 1 --2- 172.21.15.173:0/746624630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff4c802ff10 0x7ff4c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:10.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.969+0000 7ff4df7fe700 1 --2- 172.21.15.173:0/746624630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff4c802ff10 0x7ff4c80323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7ff4d00096a0 tx=0x7ff4d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:10.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:10.973+0000 7ff4dcff9700 1 -- 172.21.15.173:0/746624630 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4d40095f0 con 0x7ff4e00f5410 2024-02-19T22:06:11.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.179+0000 7ff4e6a51700 1 -- 172.21.15.173:0/746624630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4e010a3b0 con 0x7ff4e00f5410 2024-02-19T22:06:11.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.180+0000 7ff4dcff9700 1 -- 172.21.15.173:0/746624630 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff4d401db70 con 0x7ff4e00f5410 2024-02-19T22:06:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:11.180 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:11.181 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.181+0000 7ff4e6a51700 1 -- 172.21.15.173:0/746624630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff4c802ff10 msgr2=0x7ff4c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:11.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.181+0000 7ff4e6a51700 1 --2- 172.21.15.173:0/746624630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff4c802ff10 0x7ff4c80323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7ff4d00096a0 tx=0x7ff4d0006b40).stop 2024-02-19T22:06:11.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.181+0000 7ff4e6a51700 1 -- 172.21.15.173:0/746624630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e00f5410 msgr2=0x7ff4e01094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:11.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.181+0000 7ff4e6a51700 1 --2- 172.21.15.173:0/746624630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e00f5410 0x7ff4e01094e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7ff4d4005010 tx=0x7ff4d400b040).stop 2024-02-19T22:06:11.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.181+0000 7ff4e6a51700 1 -- 172.21.15.173:0/746624630 shutdown_connections 2024-02-19T22:06:11.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.181+0000 7ff4e6a51700 1 --2- 172.21.15.173:0/746624630 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7ff4c802ff10 0x7ff4c80323d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:11.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.181+0000 7ff4e6a51700 1 --2- 172.21.15.173:0/746624630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e00f5410 0x7ff4e01094e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:11.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.181+0000 7ff4e6a51700 1 -- 172.21.15.173:0/746624630 >> 172.21.15.173:0/746624630 conn(0x7ff4e00f0b70 msgr2=0x7ff4e00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:11.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.182+0000 7ff4e6a51700 1 -- 172.21.15.173:0/746624630 shutdown_connections 2024-02-19T22:06:11.182 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:11.182+0000 7ff4e6a51700 1 -- 172.21.15.173:0/746624630 wait complete. 2024-02-19T22:06:11.182 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:11.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:11 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:11.181074+0000 mon.smithi143 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.173:0/746624630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:12.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:12 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:11.346080+0000 mgr.smithi143.bissbc (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:12.982 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:12.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:13.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:13 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:13.346513+0000 mgr.smithi143.bissbc (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:14.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.938+0000 7f49456d1700 1 -- 172.21.15.173:0/1411056834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49400f51a0 msgr2=0x7f49400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:14.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.938+0000 7f49456d1700 1 --2- 172.21.15.173:0/1411056834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49400f51a0 0x7f49400f55c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f4930009fb0 tx=0x7f4930009b30).stop 2024-02-19T22:06:14.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.938+0000 7f49456d1700 1 -- 172.21.15.173:0/1411056834 shutdown_connections 2024-02-19T22:06:14.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.938+0000 7f49456d1700 1 --2- 172.21.15.173:0/1411056834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49400f51a0 0x7f49400f55c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:14.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.938+0000 7f49456d1700 1 -- 172.21.15.173:0/1411056834 >> 172.21.15.173:0/1411056834 conn(0x7f49400f09a0 msgr2=0x7f49400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:14.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.938+0000 7f49456d1700 1 -- 172.21.15.173:0/1411056834 shutdown_connections 2024-02-19T22:06:14.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.938+0000 7f49456d1700 1 -- 172.21.15.173:0/1411056834 wait complete. 2024-02-19T22:06:14.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.938+0000 7f49456d1700 1 Processor -- start 2024-02-19T22:06:14.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.939+0000 7f49456d1700 1 -- start start 2024-02-19T22:06:14.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.939+0000 7f49456d1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49400f51a0 0x7f49401054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:14.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.939+0000 7f49456d1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49401059c0 con 0x7f49400f51a0 2024-02-19T22:06:14.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.939+0000 7f493ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49400f51a0 0x7f49401054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:14.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.939+0000 7f493ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49400f51a0 0x7f49401054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:51402/0 (socket says 172.21.15.173:51402) 2024-02-19T22:06:14.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.939+0000 7f493ffff700 1 -- 172.21.15.173:0/3336616708 learned_addr learned my addr 172.21.15.173:0/3336616708 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.940+0000 7f493ffff700 1 -- 172.21.15.173:0/3336616708 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4930005040 con 0x7f49400f51a0 2024-02-19T22:06:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.940+0000 7f493ffff700 1 --2- 172.21.15.173:0/3336616708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49400f51a0 0x7f49401054d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f49300089f0 tx=0x7f493000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.940+0000 7f493cff9700 1 -- 172.21.15.173:0/3336616708 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4930009320 con 0x7f49400f51a0 2024-02-19T22:06:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.940+0000 7f49456d1700 1 -- 172.21.15.173:0/3336616708 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4940105bc0 con 0x7f49400f51a0 2024-02-19T22:06:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.940+0000 7f493cff9700 1 -- 172.21.15.173:0/3336616708 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4930013380 con 0x7f49400f51a0 2024-02-19T22:06:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.940+0000 7f493cff9700 1 -- 172.21.15.173:0/3336616708 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4930014e20 con 0x7f49400f51a0 2024-02-19T22:06:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.940+0000 7f49456d1700 1 -- 172.21.15.173:0/3336616708 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4940106050 con 0x7f49400f51a0 2024-02-19T22:06:14.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.941+0000 7f493cff9700 1 -- 172.21.15.173:0/3336616708 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4930014420 con 0x7f49400f51a0 2024-02-19T22:06:14.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.941+0000 7f49456d1700 1 -- 172.21.15.173:0/3336616708 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4940045ad0 con 0x7f49400f51a0 2024-02-19T22:06:14.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.941+0000 7f493cff9700 1 --2- 172.21.15.173:0/3336616708 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f492802ff10 0x7f49280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:14.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.941+0000 7f493cff9700 1 -- 172.21.15.173:0/3336616708 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4930040fc0 con 0x7f49400f51a0 2024-02-19T22:06:14.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.941+0000 7f493f7fe700 1 --2- 172.21.15.173:0/3336616708 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f492802ff10 0x7f49280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:14.942 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.941+0000 7f493f7fe700 1 --2- 172.21.15.173:0/3336616708 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f492802ff10 0x7f49280323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f49340096a0 tx=0x7f4934006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:14.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:14.946+0000 7f493cff9700 1 -- 172.21.15.173:0/3336616708 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f493006d2b0 con 0x7f49400f51a0 2024-02-19T22:06:15.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.149+0000 7f49456d1700 1 -- 172.21.15.173:0/3336616708 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4940106f00 con 0x7f49400f51a0 2024-02-19T22:06:15.150 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.150+0000 7f493cff9700 1 -- 172.21.15.173:0/3336616708 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f49300095f0 con 0x7f49400f51a0 2024-02-19T22:06:15.150 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:15.150 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:15.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.152+0000 7f49456d1700 1 -- 172.21.15.173:0/3336616708 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f492802ff10 msgr2=0x7f49280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:15.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.152+0000 7f49456d1700 1 --2- 172.21.15.173:0/3336616708 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f492802ff10 0x7f49280323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f49340096a0 tx=0x7f4934006b40).stop 2024-02-19T22:06:15.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.152+0000 7f49456d1700 1 -- 172.21.15.173:0/3336616708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49400f51a0 msgr2=0x7f49401054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:15.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.152+0000 7f49456d1700 1 --2- 172.21.15.173:0/3336616708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49400f51a0 0x7f49401054d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f49300089f0 tx=0x7f493000b040).stop 2024-02-19T22:06:15.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.152+0000 7f49456d1700 1 -- 172.21.15.173:0/3336616708 shutdown_connections 2024-02-19T22:06:15.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.152+0000 7f49456d1700 1 --2- 172.21.15.173:0/3336616708 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f492802ff10 0x7f49280323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:15.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.152+0000 7f49456d1700 1 --2- 172.21.15.173:0/3336616708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49400f51a0 0x7f49401054d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:15.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.152+0000 7f49456d1700 1 -- 172.21.15.173:0/3336616708 >> 172.21.15.173:0/3336616708 conn(0x7f49400f09a0 msgr2=0x7f49400f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:15.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.152+0000 7f49456d1700 1 -- 172.21.15.173:0/3336616708 shutdown_connections 2024-02-19T22:06:15.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:15.152+0000 7f49456d1700 1 -- 172.21.15.173:0/3336616708 wait complete. 2024-02-19T22:06:15.153 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:15.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:15 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:15.151242+0000 mon.smithi143 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.173:0/3336616708' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:16.661 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:16 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:15.346868+0000 mgr.smithi143.bissbc (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:16.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:16.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:18.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:18 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:17.347103+0000 mgr.smithi143.bissbc (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:18.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.930+0000 7fc8673fd700 1 -- 172.21.15.173:0/1254619446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8600f5410 msgr2=0x7fc8600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:18.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.930+0000 7fc8673fd700 1 --2- 172.21.15.173:0/1254619446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8600f5410 0x7fc8600f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fc850009fb0 tx=0x7fc850009b30).stop 2024-02-19T22:06:18.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.930+0000 7fc8673fd700 1 -- 172.21.15.173:0/1254619446 shutdown_connections 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.930+0000 7fc8673fd700 1 --2- 172.21.15.173:0/1254619446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8600f5410 0x7fc8600f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.930+0000 7fc8673fd700 1 -- 172.21.15.173:0/1254619446 >> 172.21.15.173:0/1254619446 conn(0x7fc8600f0b70 msgr2=0x7fc8600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.931+0000 7fc8673fd700 1 -- 172.21.15.173:0/1254619446 shutdown_connections 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.931+0000 7fc8673fd700 1 -- 172.21.15.173:0/1254619446 wait complete. 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.931+0000 7fc8673fd700 1 Processor -- start 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.931+0000 7fc8673fd700 1 -- start start 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.931+0000 7fc8673fd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8600f5410 0x7fc8601067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.931+0000 7fc8673fd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc860106cc0 con 0x7fc8600f5410 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.931+0000 7fc865199700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8600f5410 0x7fc8601067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.931+0000 7fc865199700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8600f5410 0x7fc8601067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:51410/0 (socket says 172.21.15.173:51410) 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.931+0000 7fc865199700 1 -- 172.21.15.173:0/2610853607 learned_addr learned my addr 172.21.15.173:0/2610853607 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.932+0000 7fc865199700 1 -- 172.21.15.173:0/2610853607 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc850005040 con 0x7fc8600f5410 2024-02-19T22:06:18.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.932+0000 7fc865199700 1 --2- 172.21.15.173:0/2610853607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8600f5410 0x7fc8601067d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fc8500089f0 tx=0x7fc85000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.932+0000 7fc855ffb700 1 -- 172.21.15.173:0/2610853607 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc850009320 con 0x7fc8600f5410 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.932+0000 7fc8673fd700 1 -- 172.21.15.173:0/2610853607 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc860106ec0 con 0x7fc8600f5410 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.932+0000 7fc855ffb700 1 -- 172.21.15.173:0/2610853607 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc850013380 con 0x7fc8600f5410 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.932+0000 7fc855ffb700 1 -- 172.21.15.173:0/2610853607 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc85001ed80 con 0x7fc8600f5410 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.932+0000 7fc8673fd700 1 -- 172.21.15.173:0/2610853607 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc860107350 con 0x7fc8600f5410 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.933+0000 7fc855ffb700 1 -- 172.21.15.173:0/2610853607 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc85001e420 con 0x7fc8600f5410 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.933+0000 7fc8673fd700 1 -- 172.21.15.173:0/2610853607 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc860045ad0 con 0x7fc8600f5410 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.933+0000 7fc855ffb700 1 --2- 172.21.15.173:0/2610853607 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc84c02ff60 0x7fc84c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.933+0000 7fc855ffb700 1 -- 172.21.15.173:0/2610853607 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8500410f0 con 0x7fc8600f5410 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.933+0000 7fc864998700 1 --2- 172.21.15.173:0/2610853607 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc84c02ff60 0x7fc84c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.934+0000 7fc864998700 1 --2- 172.21.15.173:0/2610853607 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc84c02ff60 0x7fc84c032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc85c0096a0 tx=0x7fc85c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:18.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:18.938+0000 7fc855ffb700 1 -- 172.21.15.173:0/2610853607 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc850014cc0 con 0x7fc8600f5410 2024-02-19T22:06:19.144 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.144+0000 7fc8673fd700 1 -- 172.21.15.173:0/2610853607 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc860108190 con 0x7fc8600f5410 2024-02-19T22:06:19.148 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.145+0000 7fc855ffb700 1 -- 172.21.15.173:0/2610853607 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc85002c2d0 con 0x7fc8600f5410 2024-02-19T22:06:19.148 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:19.148 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:19.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.146+0000 7fc8673fd700 1 -- 172.21.15.173:0/2610853607 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc84c02ff60 msgr2=0x7fc84c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:19.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.146+0000 7fc8673fd700 1 --2- 172.21.15.173:0/2610853607 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc84c02ff60 0x7fc84c032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc85c0096a0 tx=0x7fc85c006b40).stop 2024-02-19T22:06:19.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.146+0000 7fc8673fd700 1 -- 172.21.15.173:0/2610853607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8600f5410 msgr2=0x7fc8601067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:19.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.146+0000 7fc8673fd700 1 --2- 172.21.15.173:0/2610853607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8600f5410 0x7fc8601067d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fc8500089f0 tx=0x7fc85000b040).stop 2024-02-19T22:06:19.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.147+0000 7fc8673fd700 1 -- 172.21.15.173:0/2610853607 shutdown_connections 2024-02-19T22:06:19.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.147+0000 7fc8673fd700 1 --2- 172.21.15.173:0/2610853607 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fc84c02ff60 0x7fc84c032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:19.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.147+0000 7fc8673fd700 1 --2- 172.21.15.173:0/2610853607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8600f5410 0x7fc8601067d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:19.149 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.147+0000 7fc8673fd700 1 -- 172.21.15.173:0/2610853607 >> 172.21.15.173:0/2610853607 conn(0x7fc8600f0b70 msgr2=0x7fc86005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:19.150 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.147+0000 7fc8673fd700 1 -- 172.21.15.173:0/2610853607 shutdown_connections 2024-02-19T22:06:19.150 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:19.147+0000 7fc8673fd700 1 -- 172.21.15.173:0/2610853607 wait complete. 2024-02-19T22:06:19.150 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:19.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:19 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:19.146183+0000 mon.smithi143 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.173:0/2610853607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:20.892 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:20.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:20.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:20 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:19.347418+0000 mgr.smithi143.bissbc (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:21.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:20.575457+0000 mon.smithi143 (mon.0) 330 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/trash_purge_schedule"}]: dispatch 2024-02-19T22:06:21.911 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:21 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:20.575752+0000 mon.smithi143 (mon.0) 331 : audit [INF] from='mgr.14162 172.21.15.143:0/152121474' entity='mgr.smithi143.bissbc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi143.bissbc/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:06:22.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:22 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:21.347824+0000 mgr.smithi143.bissbc (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:23.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.015+0000 7fb79d12f700 1 -- 172.21.15.173:0/3384087423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7980f62a0 msgr2=0x7fb7980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:23.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.015+0000 7fb79d12f700 1 --2- 172.21.15.173:0/3384087423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7980f62a0 0x7fb7980f66c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fb788009fb0 tx=0x7fb788009b30).stop 2024-02-19T22:06:23.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.015+0000 7fb79d12f700 1 -- 172.21.15.173:0/3384087423 shutdown_connections 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.015+0000 7fb79d12f700 1 --2- 172.21.15.173:0/3384087423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7980f62a0 0x7fb7980f66c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.015+0000 7fb79d12f700 1 -- 172.21.15.173:0/3384087423 >> 172.21.15.173:0/3384087423 conn(0x7fb7980f1ac0 msgr2=0x7fb7980f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.015+0000 7fb79d12f700 1 -- 172.21.15.173:0/3384087423 shutdown_connections 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.015+0000 7fb79d12f700 1 -- 172.21.15.173:0/3384087423 wait complete. 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.015+0000 7fb79d12f700 1 Processor -- start 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.016+0000 7fb79d12f700 1 -- start start 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.016+0000 7fb79d12f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7980f62a0 0x7fb7981065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.016+0000 7fb79d12f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb798106ab0 con 0x7fb7980f62a0 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.016+0000 7fb797fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7980f62a0 0x7fb7981065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.016+0000 7fb797fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7980f62a0 0x7fb7981065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:40802/0 (socket says 172.21.15.173:40802) 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.016+0000 7fb797fff700 1 -- 172.21.15.173:0/2321137885 learned_addr learned my addr 172.21.15.173:0/2321137885 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.016+0000 7fb797fff700 1 -- 172.21.15.173:0/2321137885 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb788005040 con 0x7fb7980f62a0 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.017+0000 7fb797fff700 1 --2- 172.21.15.173:0/2321137885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7980f62a0 0x7fb7981065c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fb7880089f0 tx=0x7fb78800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:23.033 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.017+0000 7fb794ff9700 1 -- 172.21.15.173:0/2321137885 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb788009320 con 0x7fb7980f62a0 2024-02-19T22:06:23.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.017+0000 7fb794ff9700 1 -- 172.21.15.173:0/2321137885 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb788013380 con 0x7fb7980f62a0 2024-02-19T22:06:23.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.017+0000 7fb794ff9700 1 -- 172.21.15.173:0/2321137885 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb788014e20 con 0x7fb7980f62a0 2024-02-19T22:06:23.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.017+0000 7fb79d12f700 1 -- 172.21.15.173:0/2321137885 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb798106cb0 con 0x7fb7980f62a0 2024-02-19T22:06:23.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.017+0000 7fb79d12f700 1 -- 172.21.15.173:0/2321137885 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb798107140 con 0x7fb7980f62a0 2024-02-19T22:06:23.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.018+0000 7fb794ff9700 1 -- 172.21.15.173:0/2321137885 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7880095f0 con 0x7fb7980f62a0 2024-02-19T22:06:23.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.018+0000 7fb79d12f700 1 -- 172.21.15.173:0/2321137885 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb798045ad0 con 0x7fb7980f62a0 2024-02-19T22:06:23.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.018+0000 7fb794ff9700 1 --2- 172.21.15.173:0/2321137885 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb78002ff60 0x7fb780032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:23.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.018+0000 7fb794ff9700 1 -- 172.21.15.173:0/2321137885 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb788040e40 con 0x7fb7980f62a0 2024-02-19T22:06:23.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.018+0000 7fb7977fe700 1 --2- 172.21.15.173:0/2321137885 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb78002ff60 0x7fb780032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:23.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.018+0000 7fb7977fe700 1 --2- 172.21.15.173:0/2321137885 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb78002ff60 0x7fb780032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fb78c0096a0 tx=0x7fb78c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:23.034 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.023+0000 7fb794ff9700 1 -- 172.21.15.173:0/2321137885 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb788014420 con 0x7fb7980f62a0 2024-02-19T22:06:23.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.230+0000 7fb79d12f700 1 -- 172.21.15.173:0/2321137885 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb798107ff0 con 0x7fb7980f62a0 2024-02-19T22:06:23.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.230+0000 7fb794ff9700 1 -- 172.21.15.173:0/2321137885 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb788010230 con 0x7fb7980f62a0 2024-02-19T22:06:23.231 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:23.231 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.232+0000 7fb79d12f700 1 -- 172.21.15.173:0/2321137885 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb78002ff60 msgr2=0x7fb780032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.232+0000 7fb79d12f700 1 --2- 172.21.15.173:0/2321137885 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb78002ff60 0x7fb780032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fb78c0096a0 tx=0x7fb78c006b40).stop 2024-02-19T22:06:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.232+0000 7fb79d12f700 1 -- 172.21.15.173:0/2321137885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7980f62a0 msgr2=0x7fb7981065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.232+0000 7fb79d12f700 1 --2- 172.21.15.173:0/2321137885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7980f62a0 0x7fb7981065c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fb7880089f0 tx=0x7fb78800b040).stop 2024-02-19T22:06:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.232+0000 7fb79d12f700 1 -- 172.21.15.173:0/2321137885 shutdown_connections 2024-02-19T22:06:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.232+0000 7fb79d12f700 1 --2- 172.21.15.173:0/2321137885 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fb78002ff60 0x7fb780032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.232+0000 7fb79d12f700 1 --2- 172.21.15.173:0/2321137885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7980f62a0 0x7fb7981065c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:23.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.232+0000 7fb79d12f700 1 -- 172.21.15.173:0/2321137885 >> 172.21.15.173:0/2321137885 conn(0x7fb7980f1ac0 msgr2=0x7fb7980f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:23.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.232+0000 7fb79d12f700 1 -- 172.21.15.173:0/2321137885 shutdown_connections 2024-02-19T22:06:23.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:23.232+0000 7fb79d12f700 1 -- 172.21.15.173:0/2321137885 wait complete. 2024-02-19T22:06:23.233 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:23.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:23.231655+0000 mon.smithi143 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.173:0/2321137885' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:23.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:23 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:23.348126+0000 mgr.smithi143.bissbc (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:24.912 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:24.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:26.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:26 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:25.348477+0000 mgr.smithi143.bissbc (mgr.14162) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:26.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.946+0000 7f3440bb0700 1 -- 172.21.15.173:0/4078319059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0f5410 msgr2=0x7f343c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:26.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.946+0000 7f3440bb0700 1 --2- 172.21.15.173:0/4078319059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0f5410 0x7f343c0f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f3434009fb0 tx=0x7f3434009b30).stop 2024-02-19T22:06:26.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.946+0000 7f3440bb0700 1 -- 172.21.15.173:0/4078319059 shutdown_connections 2024-02-19T22:06:26.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.946+0000 7f3440bb0700 1 --2- 172.21.15.173:0/4078319059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0f5410 0x7f343c0f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:26.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.946+0000 7f3440bb0700 1 -- 172.21.15.173:0/4078319059 >> 172.21.15.173:0/4078319059 conn(0x7f343c0f0b70 msgr2=0x7f343c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:26.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.946+0000 7f3440bb0700 1 -- 172.21.15.173:0/4078319059 shutdown_connections 2024-02-19T22:06:26.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.946+0000 7f3440bb0700 1 -- 172.21.15.173:0/4078319059 wait complete. 2024-02-19T22:06:26.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.947+0000 7f3440bb0700 1 Processor -- start 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.947+0000 7f3440bb0700 1 -- start start 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.947+0000 7f3440bb0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0f5410 0x7f343c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.947+0000 7f3440bb0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0fb960 con 0x7f343c0f5410 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.947+0000 7f343a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0f5410 0x7f343c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.947+0000 7f343a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0f5410 0x7f343c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:40812/0 (socket says 172.21.15.173:40812) 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.947+0000 7f343a59c700 1 -- 172.21.15.173:0/2696723849 learned_addr learned my addr 172.21.15.173:0/2696723849 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.947+0000 7f343a59c700 1 -- 172.21.15.173:0/2696723849 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3434005040 con 0x7f343c0f5410 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.947+0000 7f343a59c700 1 --2- 172.21.15.173:0/2696723849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0f5410 0x7f343c0fd4e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f3434005010 tx=0x7f343400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.948+0000 7f342affd700 1 -- 172.21.15.173:0/2696723849 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3434009320 con 0x7f343c0f5410 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.948+0000 7f3440bb0700 1 -- 172.21.15.173:0/2696723849 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c0fbb00 con 0x7f343c0f5410 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.948+0000 7f342affd700 1 -- 172.21.15.173:0/2696723849 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3434013380 con 0x7f343c0f5410 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.948+0000 7f342affd700 1 -- 172.21.15.173:0/2696723849 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3434014e50 con 0x7f343c0f5410 2024-02-19T22:06:26.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.948+0000 7f3440bb0700 1 -- 172.21.15.173:0/2696723849 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c0fbf90 con 0x7f343c0f5410 2024-02-19T22:06:26.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.948+0000 7f342affd700 1 -- 172.21.15.173:0/2696723849 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34340095f0 con 0x7f343c0f5410 2024-02-19T22:06:26.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.948+0000 7f3440bb0700 1 -- 172.21.15.173:0/2696723849 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c059100 con 0x7f343c0f5410 2024-02-19T22:06:26.950 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.949+0000 7f342affd700 1 --2- 172.21.15.173:0/2696723849 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f342402ff10 0x7f34240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:26.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.949+0000 7f342affd700 1 -- 172.21.15.173:0/2696723849 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3434040f60 con 0x7f343c0f5410 2024-02-19T22:06:26.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.949+0000 7f3439d9b700 1 --2- 172.21.15.173:0/2696723849 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f342402ff10 0x7f34240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:26.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.949+0000 7f3439d9b700 1 --2- 172.21.15.173:0/2696723849 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f342402ff10 0x7f34240323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f34300096a0 tx=0x7f3430006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:26.953 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:26.954+0000 7f342affd700 1 -- 172.21.15.173:0/2696723849 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3434014420 con 0x7f343c0f5410 2024-02-19T22:06:27.160 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.160+0000 7f3440bb0700 1 -- 172.21.15.173:0/2696723849 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f343c0fafa0 con 0x7f343c0f5410 2024-02-19T22:06:27.161 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.161+0000 7f342affd700 1 -- 172.21.15.173:0/2696723849 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3434010230 con 0x7f343c0f5410 2024-02-19T22:06:27.161 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:27.161 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:27.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.162+0000 7f3440bb0700 1 -- 172.21.15.173:0/2696723849 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f342402ff10 msgr2=0x7f34240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:27.162 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.162+0000 7f3440bb0700 1 --2- 172.21.15.173:0/2696723849 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f342402ff10 0x7f34240323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f34300096a0 tx=0x7f3430006b40).stop 2024-02-19T22:06:27.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.162+0000 7f3440bb0700 1 -- 172.21.15.173:0/2696723849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0f5410 msgr2=0x7f343c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:27.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.162+0000 7f3440bb0700 1 --2- 172.21.15.173:0/2696723849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0f5410 0x7f343c0fd4e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f3434005010 tx=0x7f343400b040).stop 2024-02-19T22:06:27.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.162+0000 7f3440bb0700 1 -- 172.21.15.173:0/2696723849 shutdown_connections 2024-02-19T22:06:27.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.162+0000 7f3440bb0700 1 --2- 172.21.15.173:0/2696723849 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f342402ff10 0x7f34240323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:27.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.162+0000 7f3440bb0700 1 --2- 172.21.15.173:0/2696723849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c0f5410 0x7f343c0fd4e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:27.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.162+0000 7f3440bb0700 1 -- 172.21.15.173:0/2696723849 >> 172.21.15.173:0/2696723849 conn(0x7f343c0f0b70 msgr2=0x7f343c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:27.163 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.162+0000 7f3440bb0700 1 -- 172.21.15.173:0/2696723849 shutdown_connections 2024-02-19T22:06:27.164 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:27.163+0000 7f3440bb0700 1 -- 172.21.15.173:0/2696723849 wait complete. 2024-02-19T22:06:27.164 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:27.660 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:27 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:27.161960+0000 mon.smithi143 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.173:0/2696723849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:28.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:28.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:28.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:28 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:27.348793+0000 mgr.smithi143.bissbc (mgr.14162) 374 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:30.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.821+0000 7f1c33e78700 1 -- 172.21.15.173:0/852899268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c2c0f5410 msgr2=0x7f1c2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.821+0000 7f1c33e78700 1 --2- 172.21.15.173:0/852899268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c2c0f5410 0x7f1c2c0f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f1c1c009fb0 tx=0x7f1c1c009b30).stop 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.822+0000 7f1c33e78700 1 -- 172.21.15.173:0/852899268 shutdown_connections 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.822+0000 7f1c33e78700 1 --2- 172.21.15.173:0/852899268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c2c0f5410 0x7f1c2c0f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.822+0000 7f1c33e78700 1 -- 172.21.15.173:0/852899268 >> 172.21.15.173:0/852899268 conn(0x7f1c2c0f0b70 msgr2=0x7f1c2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.822+0000 7f1c33e78700 1 -- 172.21.15.173:0/852899268 shutdown_connections 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.822+0000 7f1c33e78700 1 -- 172.21.15.173:0/852899268 wait complete. 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.822+0000 7f1c33e78700 1 Processor -- start 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.822+0000 7f1c33e78700 1 -- start start 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.822+0000 7f1c33e78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c2c0f5410 0x7f1c2c105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.822+0000 7f1c33e78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c105c10 con 0x7f1c2c0f5410 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.823+0000 7f1c31c14700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c2c0f5410 0x7f1c2c105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.823+0000 7f1c31c14700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c2c0f5410 0x7f1c2c105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:40826/0 (socket says 172.21.15.173:40826) 2024-02-19T22:06:30.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.823+0000 7f1c31c14700 1 -- 172.21.15.173:0/3572158794 learned_addr learned my addr 172.21.15.173:0/3572158794 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:30.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.823+0000 7f1c31c14700 1 -- 172.21.15.173:0/3572158794 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1c005040 con 0x7f1c2c0f5410 2024-02-19T22:06:30.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.823+0000 7f1c31c14700 1 --2- 172.21.15.173:0/3572158794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c2c0f5410 0x7f1c2c105720 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f1c1c005010 tx=0x7f1c1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:30.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.823+0000 7f1c227fc700 1 -- 172.21.15.173:0/3572158794 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c1c009320 con 0x7f1c2c0f5410 2024-02-19T22:06:30.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.823+0000 7f1c227fc700 1 -- 172.21.15.173:0/3572158794 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c1c013380 con 0x7f1c2c0f5410 2024-02-19T22:06:30.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.823+0000 7f1c33e78700 1 -- 172.21.15.173:0/3572158794 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c105e10 con 0x7f1c2c0f5410 2024-02-19T22:06:30.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.823+0000 7f1c227fc700 1 -- 172.21.15.173:0/3572158794 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c1c014e50 con 0x7f1c2c0f5410 2024-02-19T22:06:30.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.823+0000 7f1c33e78700 1 -- 172.21.15.173:0/3572158794 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c1062a0 con 0x7f1c2c0f5410 2024-02-19T22:06:30.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.824+0000 7f1c227fc700 1 -- 172.21.15.173:0/3572158794 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c1c0095f0 con 0x7f1c2c0f5410 2024-02-19T22:06:30.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.824+0000 7f1c33e78700 1 -- 172.21.15.173:0/3572158794 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2c045ad0 con 0x7f1c2c0f5410 2024-02-19T22:06:30.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.824+0000 7f1c227fc700 1 --2- 172.21.15.173:0/3572158794 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c1802ff10 0x7f1c180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:30.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.824+0000 7f1c227fc700 1 -- 172.21.15.173:0/3572158794 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c1c040ed0 con 0x7f1c2c0f5410 2024-02-19T22:06:30.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.824+0000 7f1c31413700 1 --2- 172.21.15.173:0/3572158794 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c1802ff10 0x7f1c180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:30.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.825+0000 7f1c31413700 1 --2- 172.21.15.173:0/3572158794 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c1802ff10 0x7f1c180323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f1c280096a0 tx=0x7f1c28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:30.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:30.829+0000 7f1c227fc700 1 -- 172.21.15.173:0/3572158794 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c1c014420 con 0x7f1c2c0f5410 2024-02-19T22:06:30.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:30 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:29.349134+0000 mgr.smithi143.bissbc (mgr.14162) 375 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:31.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.037+0000 7f1c33e78700 1 -- 172.21.15.173:0/3572158794 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c2c107150 con 0x7f1c2c0f5410 2024-02-19T22:06:31.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.038+0000 7f1c227fc700 1 -- 172.21.15.173:0/3572158794 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1c1c010230 con 0x7f1c2c0f5410 2024-02-19T22:06:31.039 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:31.039 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:31.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.039+0000 7f1c33e78700 1 -- 172.21.15.173:0/3572158794 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c1802ff10 msgr2=0x7f1c180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:31.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.039+0000 7f1c33e78700 1 --2- 172.21.15.173:0/3572158794 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c1802ff10 0x7f1c180323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f1c280096a0 tx=0x7f1c28006b40).stop 2024-02-19T22:06:31.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.039+0000 7f1c33e78700 1 -- 172.21.15.173:0/3572158794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c2c0f5410 msgr2=0x7f1c2c105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:31.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.039+0000 7f1c33e78700 1 --2- 172.21.15.173:0/3572158794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c2c0f5410 0x7f1c2c105720 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f1c1c005010 tx=0x7f1c1c00b040).stop 2024-02-19T22:06:31.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.039+0000 7f1c33e78700 1 -- 172.21.15.173:0/3572158794 shutdown_connections 2024-02-19T22:06:31.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.039+0000 7f1c33e78700 1 --2- 172.21.15.173:0/3572158794 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f1c1802ff10 0x7f1c180323d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:31.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.039+0000 7f1c33e78700 1 --2- 172.21.15.173:0/3572158794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c2c0f5410 0x7f1c2c105720 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:31.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.039+0000 7f1c33e78700 1 -- 172.21.15.173:0/3572158794 >> 172.21.15.173:0/3572158794 conn(0x7f1c2c0f0b70 msgr2=0x7f1c2c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:31.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.039+0000 7f1c33e78700 1 -- 172.21.15.173:0/3572158794 shutdown_connections 2024-02-19T22:06:31.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:31.039+0000 7f1c33e78700 1 -- 172.21.15.173:0/3572158794 wait complete. 2024-02-19T22:06:31.041 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:31.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:31 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:31.038936+0000 mon.smithi143 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.173:0/3572158794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:32.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:32.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:32.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:32 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:31.349466+0000 mgr.smithi143.bissbc (mgr.14162) 376 : cluster [DBG] pgmap v360: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:34.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.788+0000 7f4a74380700 1 -- 172.21.15.173:0/2910503076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a6c0f6a80 msgr2=0x7f4a6c0f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:34.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.788+0000 7f4a74380700 1 --2- 172.21.15.173:0/2910503076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a6c0f6a80 0x7f4a6c0f6ea0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f4a5c009fb0 tx=0x7f4a5c009b30).stop 2024-02-19T22:06:34.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.788+0000 7f4a74380700 1 -- 172.21.15.173:0/2910503076 shutdown_connections 2024-02-19T22:06:34.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.788+0000 7f4a74380700 1 --2- 172.21.15.173:0/2910503076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a6c0f6a80 0x7f4a6c0f6ea0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:34.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.788+0000 7f4a74380700 1 -- 172.21.15.173:0/2910503076 >> 172.21.15.173:0/2910503076 conn(0x7f4a6c0f02e0 msgr2=0x7f4a6c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.788+0000 7f4a74380700 1 -- 172.21.15.173:0/2910503076 shutdown_connections 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.788+0000 7f4a74380700 1 -- 172.21.15.173:0/2910503076 wait complete. 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.789+0000 7f4a74380700 1 Processor -- start 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.789+0000 7f4a74380700 1 -- start start 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.789+0000 7f4a74380700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a6c0f6a80 0x7f4a6c1094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.789+0000 7f4a74380700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c0f5330 con 0x7f4a6c0f6a80 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.789+0000 7f4a7211c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a6c0f6a80 0x7f4a6c1094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.789+0000 7f4a7211c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a6c0f6a80 0x7f4a6c1094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:33828/0 (socket says 172.21.15.173:33828) 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.789+0000 7f4a7211c700 1 -- 172.21.15.173:0/1958807392 learned_addr learned my addr 172.21.15.173:0/1958807392 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.790+0000 7f4a7211c700 1 -- 172.21.15.173:0/1958807392 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c005040 con 0x7f4a6c0f6a80 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.790+0000 7f4a7211c700 1 --2- 172.21.15.173:0/1958807392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a6c0f6a80 0x7f4a6c1094a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f4a5c0089f0 tx=0x7f4a5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:34.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.790+0000 7f4a62ffd700 1 -- 172.21.15.173:0/1958807392 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a5c009320 con 0x7f4a6c0f6a80 2024-02-19T22:06:34.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.790+0000 7f4a74380700 1 -- 172.21.15.173:0/1958807392 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6c0f5530 con 0x7f4a6c0f6a80 2024-02-19T22:06:34.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.790+0000 7f4a62ffd700 1 -- 172.21.15.173:0/1958807392 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a5c013380 con 0x7f4a6c0f6a80 2024-02-19T22:06:34.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.790+0000 7f4a62ffd700 1 -- 172.21.15.173:0/1958807392 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a5c014e20 con 0x7f4a6c0f6a80 2024-02-19T22:06:34.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.790+0000 7f4a74380700 1 -- 172.21.15.173:0/1958807392 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6c109b50 con 0x7f4a6c0f6a80 2024-02-19T22:06:34.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.791+0000 7f4a62ffd700 1 -- 172.21.15.173:0/1958807392 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a5c014420 con 0x7f4a6c0f6a80 2024-02-19T22:06:34.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.791+0000 7f4a74380700 1 -- 172.21.15.173:0/1958807392 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6c045ad0 con 0x7f4a6c0f6a80 2024-02-19T22:06:34.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.791+0000 7f4a62ffd700 1 --2- 172.21.15.173:0/1958807392 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4a5802ff10 0x7f4a580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:34.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.791+0000 7f4a62ffd700 1 -- 172.21.15.173:0/1958807392 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a5c040ea0 con 0x7f4a6c0f6a80 2024-02-19T22:06:34.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.791+0000 7f4a7191b700 1 --2- 172.21.15.173:0/1958807392 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4a5802ff10 0x7f4a580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:34.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.791+0000 7f4a7191b700 1 --2- 172.21.15.173:0/1958807392 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4a5802ff10 0x7f4a580323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f4a680096a0 tx=0x7f4a68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:34.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:34.796+0000 7f4a62ffd700 1 -- 172.21.15.173:0/1958807392 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a5c01d980 con 0x7f4a6c0f6a80 2024-02-19T22:06:34.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:34 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:33.349880+0000 mgr.smithi143.bissbc (mgr.14162) 377 : cluster [DBG] pgmap v361: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:35.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.005+0000 7f4a74380700 1 -- 172.21.15.173:0/1958807392 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a6c10a370 con 0x7f4a6c0f6a80 2024-02-19T22:06:35.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.006+0000 7f4a62ffd700 1 -- 172.21.15.173:0/1958807392 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4a5c010230 con 0x7f4a6c0f6a80 2024-02-19T22:06:35.006 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:35.006 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:35.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.007+0000 7f4a74380700 1 -- 172.21.15.173:0/1958807392 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4a5802ff10 msgr2=0x7f4a580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:35.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.007+0000 7f4a74380700 1 --2- 172.21.15.173:0/1958807392 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4a5802ff10 0x7f4a580323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f4a680096a0 tx=0x7f4a68006b40).stop 2024-02-19T22:06:35.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.007+0000 7f4a74380700 1 -- 172.21.15.173:0/1958807392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a6c0f6a80 msgr2=0x7f4a6c1094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:35.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.007+0000 7f4a74380700 1 --2- 172.21.15.173:0/1958807392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a6c0f6a80 0x7f4a6c1094a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f4a5c0089f0 tx=0x7f4a5c00b040).stop 2024-02-19T22:06:35.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.007+0000 7f4a74380700 1 -- 172.21.15.173:0/1958807392 shutdown_connections 2024-02-19T22:06:35.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.007+0000 7f4a74380700 1 --2- 172.21.15.173:0/1958807392 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f4a5802ff10 0x7f4a580323d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:35.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.007+0000 7f4a74380700 1 --2- 172.21.15.173:0/1958807392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a6c0f6a80 0x7f4a6c1094a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:35.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.007+0000 7f4a74380700 1 -- 172.21.15.173:0/1958807392 >> 172.21.15.173:0/1958807392 conn(0x7f4a6c0f02e0 msgr2=0x7f4a6c0f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:35.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.007+0000 7f4a74380700 1 -- 172.21.15.173:0/1958807392 shutdown_connections 2024-02-19T22:06:35.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:35.007+0000 7f4a74380700 1 -- 172.21.15.173:0/1958807392 wait complete. 2024-02-19T22:06:35.008 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:35.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:35 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:35.006779+0000 mon.smithi143 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.173:0/1958807392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:36.808 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:36.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:36.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:36 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:35.350165+0000 mgr.smithi143.bissbc (mgr.14162) 378 : cluster [DBG] pgmap v362: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:38.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.775+0000 7fe4fbfff700 1 -- 172.21.15.173:0/297585005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc0f5810 msgr2=0x7fe4fc0f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:38.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.775+0000 7fe4fbfff700 1 --2- 172.21.15.173:0/297585005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc0f5810 0x7fe4fc0f5c30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fe4ec009fb0 tx=0x7fe4ec009b30).stop 2024-02-19T22:06:38.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.775+0000 7fe4fbfff700 1 -- 172.21.15.173:0/297585005 shutdown_connections 2024-02-19T22:06:38.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.775+0000 7fe4fbfff700 1 --2- 172.21.15.173:0/297585005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc0f5810 0x7fe4fc0f5c30 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:38.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.775+0000 7fe4fbfff700 1 -- 172.21.15.173:0/297585005 >> 172.21.15.173:0/297585005 conn(0x7fe4fc0f0120 msgr2=0x7fe4fc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:38.775 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.775+0000 7fe4fbfff700 1 -- 172.21.15.173:0/297585005 shutdown_connections 2024-02-19T22:06:38.776 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.775+0000 7fe4fbfff700 1 -- 172.21.15.173:0/297585005 wait complete. 2024-02-19T22:06:38.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.776+0000 7fe4fbfff700 1 Processor -- start 2024-02-19T22:06:38.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.776+0000 7fe4fbfff700 1 -- start start 2024-02-19T22:06:38.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.776+0000 7fe4fbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc0f5810 0x7fe4fc105470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:38.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.776+0000 7fe4fbfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc105960 con 0x7fe4fc0f5810 2024-02-19T22:06:38.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.776+0000 7fe4faffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc0f5810 0x7fe4fc105470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:38.777 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.776+0000 7fe4faffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc0f5810 0x7fe4fc105470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:33852/0 (socket says 172.21.15.173:33852) 2024-02-19T22:06:38.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.776+0000 7fe4faffd700 1 -- 172.21.15.173:0/3780787538 learned_addr learned my addr 172.21.15.173:0/3780787538 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:38.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.776+0000 7fe4faffd700 1 -- 172.21.15.173:0/3780787538 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec005040 con 0x7fe4fc0f5810 2024-02-19T22:06:38.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.777+0000 7fe4faffd700 1 --2- 172.21.15.173:0/3780787538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc0f5810 0x7fe4fc105470 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fe4ec007a10 tx=0x7fe4ec009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:38.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.777+0000 7fe4e37fe700 1 -- 172.21.15.173:0/3780787538 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4ec018070 con 0x7fe4fc0f5810 2024-02-19T22:06:38.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.777+0000 7fe4fbfff700 1 -- 172.21.15.173:0/3780787538 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4fc105b60 con 0x7fe4fc0f5810 2024-02-19T22:06:38.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.777+0000 7fe4e37fe700 1 -- 172.21.15.173:0/3780787538 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4ec013410 con 0x7fe4fc0f5810 2024-02-19T22:06:38.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.777+0000 7fe4e37fe700 1 -- 172.21.15.173:0/3780787538 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4ec01c3c0 con 0x7fe4fc0f5810 2024-02-19T22:06:38.778 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.777+0000 7fe4fbfff700 1 -- 172.21.15.173:0/3780787538 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4fc105ff0 con 0x7fe4fc0f5810 2024-02-19T22:06:38.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.778+0000 7fe4e37fe700 1 -- 172.21.15.173:0/3780787538 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4ec013ca0 con 0x7fe4fc0f5810 2024-02-19T22:06:38.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.778+0000 7fe4fbfff700 1 -- 172.21.15.173:0/3780787538 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4fc045ad0 con 0x7fe4fc0f5810 2024-02-19T22:06:38.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.778+0000 7fe4e37fe700 1 --2- 172.21.15.173:0/3780787538 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe4e402ff60 0x7fe4e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:38.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.778+0000 7fe4e37fe700 1 -- 172.21.15.173:0/3780787538 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4ec0414b0 con 0x7fe4fc0f5810 2024-02-19T22:06:38.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.778+0000 7fe4fa7fc700 1 --2- 172.21.15.173:0/3780787538 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe4e402ff60 0x7fe4e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:38.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.778+0000 7fe4fa7fc700 1 --2- 172.21.15.173:0/3780787538 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe4e402ff60 0x7fe4e4032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe4f00096a0 tx=0x7fe4f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:38.783 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.783+0000 7fe4e37fe700 1 -- 172.21.15.173:0/3780787538 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4ec01d420 con 0x7fe4fc0f5810 2024-02-19T22:06:38.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:38 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:37.350446+0000 mgr.smithi143.bissbc (mgr.14162) 379 : cluster [DBG] pgmap v363: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:38.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.986+0000 7fe4fbfff700 1 -- 172.21.15.173:0/3780787538 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4fc106e40 con 0x7fe4fc0f5810 2024-02-19T22:06:38.986 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.986+0000 7fe4e37fe700 1 -- 172.21.15.173:0/3780787538 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe4ec025ce0 con 0x7fe4fc0f5810 2024-02-19T22:06:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:38.987 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:38.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.988+0000 7fe4fbfff700 1 -- 172.21.15.173:0/3780787538 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe4e402ff60 msgr2=0x7fe4e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:38.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.988+0000 7fe4fbfff700 1 --2- 172.21.15.173:0/3780787538 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe4e402ff60 0x7fe4e4032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe4f00096a0 tx=0x7fe4f0006b40).stop 2024-02-19T22:06:38.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.988+0000 7fe4fbfff700 1 -- 172.21.15.173:0/3780787538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc0f5810 msgr2=0x7fe4fc105470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:38.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.988+0000 7fe4fbfff700 1 --2- 172.21.15.173:0/3780787538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc0f5810 0x7fe4fc105470 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fe4ec007a10 tx=0x7fe4ec009b30).stop 2024-02-19T22:06:38.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.988+0000 7fe4fbfff700 1 -- 172.21.15.173:0/3780787538 shutdown_connections 2024-02-19T22:06:38.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.988+0000 7fe4fbfff700 1 --2- 172.21.15.173:0/3780787538 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe4e402ff60 0x7fe4e4032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:38.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.988+0000 7fe4fbfff700 1 --2- 172.21.15.173:0/3780787538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc0f5810 0x7fe4fc105470 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:38.988 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.988+0000 7fe4fbfff700 1 -- 172.21.15.173:0/3780787538 >> 172.21.15.173:0/3780787538 conn(0x7fe4fc0f0120 msgr2=0x7fe4fc0f0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:38.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.988+0000 7fe4fbfff700 1 -- 172.21.15.173:0/3780787538 shutdown_connections 2024-02-19T22:06:38.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:38.988+0000 7fe4fbfff700 1 -- 172.21.15.173:0/3780787538 wait complete. 2024-02-19T22:06:38.989 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:39.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:39 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:38.987682+0000 mon.smithi143 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.173:0/3780787538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:40.728 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:40.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:40.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:40 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:39.350829+0000 mgr.smithi143.bissbc (mgr.14162) 380 : cluster [DBG] pgmap v364: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:42.725 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.713+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/2692245289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a00f6510 msgr2=0x7fe5a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:42.725 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.713+0000 7fe5a6dfe700 1 --2- 172.21.15.173:0/2692245289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a00f6510 0x7fe5a00f6930 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fe594009fb0 tx=0x7fe594009b30).stop 2024-02-19T22:06:42.725 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.713+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/2692245289 shutdown_connections 2024-02-19T22:06:42.725 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.713+0000 7fe5a6dfe700 1 --2- 172.21.15.173:0/2692245289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a00f6510 0x7fe5a00f6930 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:42.725 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.713+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/2692245289 >> 172.21.15.173:0/2692245289 conn(0x7fe5a00f1c70 msgr2=0x7fe5a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:42.725 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.713+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/2692245289 shutdown_connections 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.713+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/2692245289 wait complete. 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.714+0000 7fe5a6dfe700 1 Processor -- start 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.714+0000 7fe5a6dfe700 1 -- start start 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.714+0000 7fe5a6dfe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a00f6510 0x7fe5a006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.714+0000 7fe5a6dfe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a0069e20 con 0x7fe5a00f6510 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.714+0000 7fe5a4b9a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a00f6510 0x7fe5a006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.714+0000 7fe5a4b9a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a00f6510 0x7fe5a006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:55510/0 (socket says 172.21.15.173:55510) 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.714+0000 7fe5a4b9a700 1 -- 172.21.15.173:0/3771173238 learned_addr learned my addr 172.21.15.173:0/3771173238 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.715+0000 7fe5a4b9a700 1 -- 172.21.15.173:0/3771173238 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe594005040 con 0x7fe5a00f6510 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.715+0000 7fe5a4b9a700 1 --2- 172.21.15.173:0/3771173238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a00f6510 0x7fe5a006b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe594005010 tx=0x7fe59400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.715+0000 7fe59d7fa700 1 -- 172.21.15.173:0/3771173238 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe594009320 con 0x7fe5a00f6510 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.715+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/3771173238 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a0069fc0 con 0x7fe5a00f6510 2024-02-19T22:06:42.726 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.715+0000 7fe59d7fa700 1 -- 172.21.15.173:0/3771173238 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe594013380 con 0x7fe5a00f6510 2024-02-19T22:06:42.727 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.715+0000 7fe59d7fa700 1 -- 172.21.15.173:0/3771173238 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe594014e50 con 0x7fe5a00f6510 2024-02-19T22:06:42.727 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.715+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/3771173238 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a006a450 con 0x7fe5a00f6510 2024-02-19T22:06:42.727 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.716+0000 7fe59d7fa700 1 -- 172.21.15.173:0/3771173238 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5940095f0 con 0x7fe5a00f6510 2024-02-19T22:06:42.727 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.716+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/3771173238 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a0059100 con 0x7fe5a00f6510 2024-02-19T22:06:42.727 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.716+0000 7fe59d7fa700 1 --2- 172.21.15.173:0/3771173238 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe59002ff10 0x7fe5900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:42.727 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.716+0000 7fe59d7fa700 1 -- 172.21.15.173:0/3771173238 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe594040e90 con 0x7fe5a00f6510 2024-02-19T22:06:42.727 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.716+0000 7fe59ffff700 1 --2- 172.21.15.173:0/3771173238 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe59002ff10 0x7fe5900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:42.727 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.717+0000 7fe59ffff700 1 --2- 172.21.15.173:0/3771173238 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe59002ff10 0x7fe5900323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe58c0096a0 tx=0x7fe58c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:42.727 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.721+0000 7fe59d7fa700 1 -- 172.21.15.173:0/3771173238 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe594014420 con 0x7fe5a00f6510 2024-02-19T22:06:42.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:42 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:41.351168+0000 mgr.smithi143.bissbc (mgr.14162) 381 : cluster [DBG] pgmap v365: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:42.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.927+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/3771173238 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5a00fc0a0 con 0x7fe5a00f6510 2024-02-19T22:06:42.928 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.928+0000 7fe59d7fa700 1 -- 172.21.15.173:0/3771173238 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe594010230 con 0x7fe5a00f6510 2024-02-19T22:06:42.928 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:42.928 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:42.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.930+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/3771173238 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe59002ff10 msgr2=0x7fe5900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:42.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.930+0000 7fe5a6dfe700 1 --2- 172.21.15.173:0/3771173238 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe59002ff10 0x7fe5900323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe58c0096a0 tx=0x7fe58c006b40).stop 2024-02-19T22:06:42.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.930+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/3771173238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a00f6510 msgr2=0x7fe5a006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:42.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.930+0000 7fe5a6dfe700 1 --2- 172.21.15.173:0/3771173238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a00f6510 0x7fe5a006b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe594005010 tx=0x7fe59400b040).stop 2024-02-19T22:06:42.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.930+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/3771173238 shutdown_connections 2024-02-19T22:06:42.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.930+0000 7fe5a6dfe700 1 --2- 172.21.15.173:0/3771173238 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fe59002ff10 0x7fe5900323d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:42.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.930+0000 7fe5a6dfe700 1 --2- 172.21.15.173:0/3771173238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a00f6510 0x7fe5a006b9a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:42.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.930+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/3771173238 >> 172.21.15.173:0/3771173238 conn(0x7fe5a00f1c70 msgr2=0x7fe5a00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:42.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.930+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/3771173238 shutdown_connections 2024-02-19T22:06:42.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:42.930+0000 7fe5a6dfe700 1 -- 172.21.15.173:0/3771173238 wait complete. 2024-02-19T22:06:42.931 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:43.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:43 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:42.929011+0000 mon.smithi143 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.173:0/3771173238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:44.692 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:44.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:44.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:44 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:43.351475+0000 mgr.smithi143.bissbc (mgr.14162) 382 : cluster [DBG] pgmap v366: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:46.781 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.779+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/4017143040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a940f6980 msgr2=0x7f8a940f6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:46.781 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.779+0000 7f8a9bf8a700 1 --2- 172.21.15.173:0/4017143040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a940f6980 0x7f8a940f6da0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f8a84009fb0 tx=0x7f8a84009b30).stop 2024-02-19T22:06:46.781 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.779+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/4017143040 shutdown_connections 2024-02-19T22:06:46.781 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.779+0000 7f8a9bf8a700 1 --2- 172.21.15.173:0/4017143040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a940f6980 0x7f8a940f6da0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:46.781 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.779+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/4017143040 >> 172.21.15.173:0/4017143040 conn(0x7f8a940f0260 msgr2=0x7f8a940f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:46.781 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.779+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/4017143040 shutdown_connections 2024-02-19T22:06:46.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.779+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/4017143040 wait complete. 2024-02-19T22:06:46.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.780+0000 7f8a9bf8a700 1 Processor -- start 2024-02-19T22:06:46.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.780+0000 7f8a9bf8a700 1 -- start start 2024-02-19T22:06:46.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.780+0000 7f8a9bf8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a940f6980 0x7f8a941056b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:46.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.780+0000 7f8a9bf8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a94105ba0 con 0x7f8a940f6980 2024-02-19T22:06:46.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.780+0000 7f8a99d26700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a940f6980 0x7f8a941056b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:46.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.780+0000 7f8a99d26700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a940f6980 0x7f8a941056b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:55532/0 (socket says 172.21.15.173:55532) 2024-02-19T22:06:46.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.780+0000 7f8a99d26700 1 -- 172.21.15.173:0/3717458935 learned_addr learned my addr 172.21.15.173:0/3717458935 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:46.783 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.780+0000 7f8a99d26700 1 -- 172.21.15.173:0/3717458935 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a84005040 con 0x7f8a940f6980 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.781+0000 7f8a99d26700 1 --2- 172.21.15.173:0/3717458935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a940f6980 0x7f8a941056b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f8a84005010 tx=0x7f8a8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.781+0000 7f8a8a7fc700 1 -- 172.21.15.173:0/3717458935 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a84009320 con 0x7f8a940f6980 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.781+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/3717458935 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a94105da0 con 0x7f8a940f6980 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.781+0000 7f8a8a7fc700 1 -- 172.21.15.173:0/3717458935 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a84013380 con 0x7f8a940f6980 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.781+0000 7f8a8a7fc700 1 -- 172.21.15.173:0/3717458935 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a84014e50 con 0x7f8a940f6980 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.781+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/3717458935 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a94106230 con 0x7f8a940f6980 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.782+0000 7f8a8a7fc700 1 -- 172.21.15.173:0/3717458935 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a84014420 con 0x7f8a940f6980 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.782+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/3717458935 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a94045ad0 con 0x7f8a940f6980 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.782+0000 7f8a8a7fc700 1 --2- 172.21.15.173:0/3717458935 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a8002ff10 0x7f8a800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.782+0000 7f8a8a7fc700 1 -- 172.21.15.173:0/3717458935 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a84040e30 con 0x7f8a940f6980 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.782+0000 7f8a99525700 1 --2- 172.21.15.173:0/3717458935 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a8002ff10 0x7f8a800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:46.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.783+0000 7f8a99525700 1 --2- 172.21.15.173:0/3717458935 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a8002ff10 0x7f8a800323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8a900096a0 tx=0x7f8a90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:46.788 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.788+0000 7f8a8a7fc700 1 -- 172.21.15.173:0/3717458935 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a840095f0 con 0x7f8a940f6980 2024-02-19T22:06:46.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:46 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:45.351818+0000 mgr.smithi143.bissbc (mgr.14162) 383 : cluster [DBG] pgmap v367: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:46.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.993+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/3717458935 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a941070e0 con 0x7f8a940f6980 2024-02-19T22:06:46.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.994+0000 7f8a8a7fc700 1 -- 172.21.15.173:0/3717458935 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8a8401db70 con 0x7f8a940f6980 2024-02-19T22:06:46.995 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:46.995 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:46.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.996+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/3717458935 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a8002ff10 msgr2=0x7f8a800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:46.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.996+0000 7f8a9bf8a700 1 --2- 172.21.15.173:0/3717458935 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a8002ff10 0x7f8a800323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8a900096a0 tx=0x7f8a90006b40).stop 2024-02-19T22:06:46.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.996+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/3717458935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a940f6980 msgr2=0x7f8a941056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:46.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.996+0000 7f8a9bf8a700 1 --2- 172.21.15.173:0/3717458935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a940f6980 0x7f8a941056b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f8a84005010 tx=0x7f8a8400b040).stop 2024-02-19T22:06:46.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.996+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/3717458935 shutdown_connections 2024-02-19T22:06:46.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.996+0000 7f8a9bf8a700 1 --2- 172.21.15.173:0/3717458935 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f8a8002ff10 0x7f8a800323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:46.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.996+0000 7f8a9bf8a700 1 --2- 172.21.15.173:0/3717458935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a940f6980 0x7f8a941056b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:46.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.996+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/3717458935 >> 172.21.15.173:0/3717458935 conn(0x7f8a940f0260 msgr2=0x7f8a940f0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:46.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.996+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/3717458935 shutdown_connections 2024-02-19T22:06:46.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:46.996+0000 7f8a9bf8a700 1 -- 172.21.15.173:0/3717458935 wait complete. 2024-02-19T22:06:46.998 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:47.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:47 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:46.996022+0000 mon.smithi143 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.173:0/3717458935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:48.745 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:48.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:48.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:48 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:47.352143+0000 mgr.smithi143.bissbc (mgr.14162) 384 : cluster [DBG] pgmap v368: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:49.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:49 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:49.352528+0000 mgr.smithi143.bissbc (mgr.14162) 385 : cluster [DBG] pgmap v369: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:50.754 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.752+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/58705968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c40f5410 msgr2=0x7fa9c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:50.754 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.752+0000 7fa9ca0a3700 1 --2- 172.21.15.173:0/58705968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c40f5410 0x7fa9c40f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fa9b8009fb0 tx=0x7fa9b8009b30).stop 2024-02-19T22:06:50.754 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.752+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/58705968 shutdown_connections 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.752+0000 7fa9ca0a3700 1 --2- 172.21.15.173:0/58705968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c40f5410 0x7fa9c40f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.752+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/58705968 >> 172.21.15.173:0/58705968 conn(0x7fa9c40f0b70 msgr2=0x7fa9c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.752+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/58705968 shutdown_connections 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.752+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/58705968 wait complete. 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.753+0000 7fa9ca0a3700 1 Processor -- start 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.753+0000 7fa9ca0a3700 1 -- start start 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.753+0000 7fa9ca0a3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c40f5410 0x7fa9c41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.753+0000 7fa9ca0a3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c4106cc0 con 0x7fa9c40f5410 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.753+0000 7fa9c37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c40f5410 0x7fa9c41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.754+0000 7fa9c37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c40f5410 0x7fa9c41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:55548/0 (socket says 172.21.15.173:55548) 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.754+0000 7fa9c37fe700 1 -- 172.21.15.173:0/2238842169 learned_addr learned my addr 172.21.15.173:0/2238842169 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.754+0000 7fa9c37fe700 1 -- 172.21.15.173:0/2238842169 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b8005040 con 0x7fa9c40f5410 2024-02-19T22:06:50.755 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.754+0000 7fa9c37fe700 1 --2- 172.21.15.173:0/2238842169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c40f5410 0x7fa9c41067d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fa9b8005010 tx=0x7fa9b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:50.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.754+0000 7fa9abfff700 1 -- 172.21.15.173:0/2238842169 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9b8009320 con 0x7fa9c40f5410 2024-02-19T22:06:50.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.754+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/2238842169 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c4106ec0 con 0x7fa9c40f5410 2024-02-19T22:06:50.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.754+0000 7fa9abfff700 1 -- 172.21.15.173:0/2238842169 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9b8013380 con 0x7fa9c40f5410 2024-02-19T22:06:50.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.754+0000 7fa9abfff700 1 -- 172.21.15.173:0/2238842169 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9b8014e50 con 0x7fa9c40f5410 2024-02-19T22:06:50.756 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.754+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/2238842169 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c4107350 con 0x7fa9c40f5410 2024-02-19T22:06:50.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.755+0000 7fa9abfff700 1 -- 172.21.15.173:0/2238842169 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9b80263a0 con 0x7fa9c40f5410 2024-02-19T22:06:50.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.755+0000 7fa9abfff700 1 --2- 172.21.15.173:0/2238842169 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa9ac02ff10 0x7fa9ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:50.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.755+0000 7fa9abfff700 1 -- 172.21.15.173:0/2238842169 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9b8041160 con 0x7fa9c40f5410 2024-02-19T22:06:50.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.755+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/2238842169 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c4059100 con 0x7fa9c40f5410 2024-02-19T22:06:50.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.755+0000 7fa9c2ffd700 1 --2- 172.21.15.173:0/2238842169 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa9ac02ff10 0x7fa9ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:50.757 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.756+0000 7fa9c2ffd700 1 --2- 172.21.15.173:0/2238842169 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa9ac02ff10 0x7fa9ac0323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa9b40096a0 tx=0x7fa9b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:50.760 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.760+0000 7fa9abfff700 1 -- 172.21.15.173:0/2238842169 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9b8012920 con 0x7fa9c40f5410 2024-02-19T22:06:50.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.966+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/2238842169 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9c40fafa0 con 0x7fa9c40f5410 2024-02-19T22:06:50.966 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.966+0000 7fa9abfff700 1 -- 172.21.15.173:0/2238842169 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa9b8012330 con 0x7fa9c40f5410 2024-02-19T22:06:50.966 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:50.966 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:50.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.968+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/2238842169 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa9ac02ff10 msgr2=0x7fa9ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:50.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.968+0000 7fa9ca0a3700 1 --2- 172.21.15.173:0/2238842169 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa9ac02ff10 0x7fa9ac0323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa9b40096a0 tx=0x7fa9b4006b40).stop 2024-02-19T22:06:50.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.968+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/2238842169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c40f5410 msgr2=0x7fa9c41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:50.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.968+0000 7fa9ca0a3700 1 --2- 172.21.15.173:0/2238842169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c40f5410 0x7fa9c41067d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fa9b8005010 tx=0x7fa9b800b040).stop 2024-02-19T22:06:50.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.968+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/2238842169 shutdown_connections 2024-02-19T22:06:50.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.968+0000 7fa9ca0a3700 1 --2- 172.21.15.173:0/2238842169 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7fa9ac02ff10 0x7fa9ac0323d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:50.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.968+0000 7fa9ca0a3700 1 --2- 172.21.15.173:0/2238842169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c40f5410 0x7fa9c41067d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:50.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.968+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/2238842169 >> 172.21.15.173:0/2238842169 conn(0x7fa9c40f0b70 msgr2=0x7fa9c405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:50.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.968+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/2238842169 shutdown_connections 2024-02-19T22:06:50.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:50.968+0000 7fa9ca0a3700 1 -- 172.21.15.173:0/2238842169 wait complete. 2024-02-19T22:06:50.969 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:51.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:51 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:50.967386+0000 mon.smithi143 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.173:0/2238842169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:52.410 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:52 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:51.352913+0000 mgr.smithi143.bissbc (mgr.14162) 386 : cluster [DBG] pgmap v370: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:52.655 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:52.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.710+0000 7f18acccd700 1 -- 172.21.15.173:0/3221679740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18a80f5410 msgr2=0x7f18a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.710+0000 7f18acccd700 1 --2- 172.21.15.173:0/3221679740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18a80f5410 0x7f18a80f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f1898009fb0 tx=0x7f1898009b30).stop 2024-02-19T22:06:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.710+0000 7f18acccd700 1 -- 172.21.15.173:0/3221679740 shutdown_connections 2024-02-19T22:06:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.710+0000 7f18acccd700 1 --2- 172.21.15.173:0/3221679740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18a80f5410 0x7f18a80f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.710+0000 7f18acccd700 1 -- 172.21.15.173:0/3221679740 >> 172.21.15.173:0/3221679740 conn(0x7f18a80f0b70 msgr2=0x7f18a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.710+0000 7f18acccd700 1 -- 172.21.15.173:0/3221679740 shutdown_connections 2024-02-19T22:06:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.710+0000 7f18acccd700 1 -- 172.21.15.173:0/3221679740 wait complete. 2024-02-19T22:06:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.711+0000 7f18acccd700 1 Processor -- start 2024-02-19T22:06:54.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.711+0000 7f18acccd700 1 -- start start 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.711+0000 7f18acccd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18a80f5410 0x7f18a81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.711+0000 7f18acccd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a8105bd0 con 0x7f18a80f5410 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.711+0000 7f18a659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18a80f5410 0x7f18a81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.711+0000 7f18a659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18a80f5410 0x7f18a81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:53912/0 (socket says 172.21.15.173:53912) 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.711+0000 7f18a659c700 1 -- 172.21.15.173:0/996087280 learned_addr learned my addr 172.21.15.173:0/996087280 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.711+0000 7f18a659c700 1 -- 172.21.15.173:0/996087280 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1898005040 con 0x7f18a80f5410 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.711+0000 7f18a659c700 1 --2- 172.21.15.173:0/996087280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18a80f5410 0x7f18a81056e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f1898005010 tx=0x7f189800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.712+0000 7f189effd700 1 -- 172.21.15.173:0/996087280 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1898009320 con 0x7f18a80f5410 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.712+0000 7f18acccd700 1 -- 172.21.15.173:0/996087280 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a8105dd0 con 0x7f18a80f5410 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.712+0000 7f189effd700 1 -- 172.21.15.173:0/996087280 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1898013380 con 0x7f18a80f5410 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.712+0000 7f189effd700 1 -- 172.21.15.173:0/996087280 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1898014e50 con 0x7f18a80f5410 2024-02-19T22:06:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.712+0000 7f18acccd700 1 -- 172.21.15.173:0/996087280 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a8106260 con 0x7f18a80f5410 2024-02-19T22:06:54.714 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.712+0000 7f189effd700 1 -- 172.21.15.173:0/996087280 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18980095f0 con 0x7f18a80f5410 2024-02-19T22:06:54.714 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.712+0000 7f18acccd700 1 -- 172.21.15.173:0/996087280 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a8045ad0 con 0x7f18a80f5410 2024-02-19T22:06:54.714 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.713+0000 7f189effd700 1 --2- 172.21.15.173:0/996087280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f189402ff10 0x7f18940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:54.714 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.713+0000 7f189effd700 1 -- 172.21.15.173:0/996087280 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1898040e90 con 0x7f18a80f5410 2024-02-19T22:06:54.715 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.713+0000 7f18a5d9b700 1 --2- 172.21.15.173:0/996087280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f189402ff10 0x7f18940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:54.715 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.713+0000 7f18a5d9b700 1 --2- 172.21.15.173:0/996087280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f189402ff10 0x7f18940323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f18900096a0 tx=0x7f1890006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:54.718 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.718+0000 7f189effd700 1 -- 172.21.15.173:0/996087280 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1898014420 con 0x7f18a80f5410 2024-02-19T22:06:54.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:54 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:53.353187+0000 mgr.smithi143.bissbc (mgr.14162) 387 : cluster [DBG] pgmap v371: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:54.924 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.924+0000 7f18acccd700 1 -- 172.21.15.173:0/996087280 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18a8107110 con 0x7f18a80f5410 2024-02-19T22:06:54.925 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.925+0000 7f189effd700 1 -- 172.21.15.173:0/996087280 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1898010230 con 0x7f18a80f5410 2024-02-19T22:06:54.925 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:54.925 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:54.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.926+0000 7f18acccd700 1 -- 172.21.15.173:0/996087280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f189402ff10 msgr2=0x7f18940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:54.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.926+0000 7f18acccd700 1 --2- 172.21.15.173:0/996087280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f189402ff10 0x7f18940323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f18900096a0 tx=0x7f1890006b40).stop 2024-02-19T22:06:54.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.926+0000 7f18acccd700 1 -- 172.21.15.173:0/996087280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18a80f5410 msgr2=0x7f18a81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:54.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.926+0000 7f18acccd700 1 --2- 172.21.15.173:0/996087280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18a80f5410 0x7f18a81056e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f1898005010 tx=0x7f189800b040).stop 2024-02-19T22:06:54.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.926+0000 7f18acccd700 1 -- 172.21.15.173:0/996087280 shutdown_connections 2024-02-19T22:06:54.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.926+0000 7f18acccd700 1 --2- 172.21.15.173:0/996087280 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f189402ff10 0x7f18940323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:54.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.926+0000 7f18acccd700 1 --2- 172.21.15.173:0/996087280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18a80f5410 0x7f18a81056e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:54.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.926+0000 7f18acccd700 1 -- 172.21.15.173:0/996087280 >> 172.21.15.173:0/996087280 conn(0x7f18a80f0b70 msgr2=0x7f18a80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:54.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.927+0000 7f18acccd700 1 -- 172.21.15.173:0/996087280 shutdown_connections 2024-02-19T22:06:54.927 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:54.927+0000 7f18acccd700 1 -- 172.21.15.173:0/996087280 wait complete. 2024-02-19T22:06:54.928 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:55.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:55 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:54.925917+0000 mon.smithi143 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.173:0/996087280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:56.643 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:56.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38187086-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:56.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:56 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:55.353513+0000 mgr.smithi143.bissbc (mgr.14162) 388 : cluster [DBG] pgmap v372: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:58.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.675+0000 7f234c951700 1 -- 172.21.15.173:0/723189286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23480f62a0 msgr2=0x7f23480f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:58.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.675+0000 7f234c951700 1 --2- 172.21.15.173:0/723189286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23480f62a0 0x7f23480f66c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f2330009fb0 tx=0x7f2330009b30).stop 2024-02-19T22:06:58.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.675+0000 7f234c951700 1 -- 172.21.15.173:0/723189286 shutdown_connections 2024-02-19T22:06:58.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.675+0000 7f234c951700 1 --2- 172.21.15.173:0/723189286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23480f62a0 0x7f23480f66c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:58.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.675+0000 7f234c951700 1 -- 172.21.15.173:0/723189286 >> 172.21.15.173:0/723189286 conn(0x7f23480f1aa0 msgr2=0x7f23480f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:58.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.675+0000 7f234c951700 1 -- 172.21.15.173:0/723189286 shutdown_connections 2024-02-19T22:06:58.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.675+0000 7f234c951700 1 -- 172.21.15.173:0/723189286 wait complete. 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.676+0000 7f234c951700 1 Processor -- start 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.676+0000 7f234c951700 1 -- start start 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.676+0000 7f234c951700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23480f62a0 0x7f23481065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.676+0000 7f234c951700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2348106ac0 con 0x7f23480f62a0 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.676+0000 7f23477fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23480f62a0 0x7f23481065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.676+0000 7f23477fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23480f62a0 0x7f23481065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.173:53922/0 (socket says 172.21.15.173:53922) 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.676+0000 7f23477fe700 1 -- 172.21.15.173:0/4128309899 learned_addr learned my addr 172.21.15.173:0/4128309899 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.677+0000 7f23477fe700 1 -- 172.21.15.173:0/4128309899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2330005040 con 0x7f23480f62a0 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.677+0000 7f23477fe700 1 --2- 172.21.15.173:0/4128309899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23480f62a0 0x7f23481065d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f23300089f0 tx=0x7f233000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.677+0000 7f233ffff700 1 -- 172.21.15.173:0/4128309899 <== mon.0 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2330009320 con 0x7f23480f62a0 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.677+0000 7f234c951700 1 -- 172.21.15.173:0/4128309899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2348106cc0 con 0x7f23480f62a0 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.677+0000 7f233ffff700 1 -- 172.21.15.173:0/4128309899 <== mon.0 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2330013380 con 0x7f23480f62a0 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.677+0000 7f233ffff700 1 -- 172.21.15.173:0/4128309899 <== mon.0 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2330014e20 con 0x7f23480f62a0 2024-02-19T22:06:58.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.677+0000 7f234c951700 1 -- 172.21.15.173:0/4128309899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2348107150 con 0x7f23480f62a0 2024-02-19T22:06:58.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.678+0000 7f233ffff700 1 -- 172.21.15.173:0/4128309899 <== mon.0 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2330014420 con 0x7f23480f62a0 2024-02-19T22:06:58.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.678+0000 7f234c951700 1 -- 172.21.15.173:0/4128309899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2348045ad0 con 0x7f23480f62a0 2024-02-19T22:06:58.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.678+0000 7f233ffff700 1 --2- 172.21.15.173:0/4128309899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f233402ff10 0x7f23340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:58.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.678+0000 7f233ffff700 1 -- 172.21.15.173:0/4128309899 <== mon.0 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2330040f20 con 0x7f23480f62a0 2024-02-19T22:06:58.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.678+0000 7f2346ffd700 1 --2- 172.21.15.173:0/4128309899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f233402ff10 0x7f23340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:58.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.679+0000 7f2346ffd700 1 --2- 172.21.15.173:0/4128309899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f233402ff10 0x7f23340323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f23380096a0 tx=0x7f2338006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:58.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.683+0000 7f233ffff700 1 -- 172.21.15.173:0/4128309899 <== mon.0 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f233006d2b0 con 0x7f23480f62a0 2024-02-19T22:06:58.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.891+0000 7f234c951700 1 -- 172.21.15.173:0/4128309899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2348108000 con 0x7f23480f62a0 2024-02-19T22:06:58.892 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.892+0000 7f233ffff700 1 -- 172.21.15.173:0/4128309899 <== mon.0 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f23300095f0 con 0x7f23480f62a0 2024-02-19T22:06:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-02-19T22:06:58.892 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"38187086-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:53:14.854843Z","created":"2024-02-19T21:53:14.854843Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:58.893 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.893+0000 7f234c951700 1 -- 172.21.15.173:0/4128309899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f233402ff10 msgr2=0x7f23340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:58.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.893+0000 7f234c951700 1 --2- 172.21.15.173:0/4128309899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f233402ff10 0x7f23340323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f23380096a0 tx=0x7f2338006b40).stop 2024-02-19T22:06:58.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.893+0000 7f234c951700 1 -- 172.21.15.173:0/4128309899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23480f62a0 msgr2=0x7f23481065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:58.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.893+0000 7f234c951700 1 --2- 172.21.15.173:0/4128309899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23480f62a0 0x7f23481065d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f23300089f0 tx=0x7f233000b040).stop 2024-02-19T22:06:58.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.893+0000 7f234c951700 1 -- 172.21.15.173:0/4128309899 shutdown_connections 2024-02-19T22:06:58.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.893+0000 7f234c951700 1 --2- 172.21.15.173:0/4128309899 >> [v2:172.21.15.143:6800/2497069397,v1:172.21.15.143:6801/2497069397] conn(0x7f233402ff10 0x7f23340323d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:58.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.893+0000 7f234c951700 1 --2- 172.21.15.173:0/4128309899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23480f62a0 0x7f23481065d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:58.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.893+0000 7f234c951700 1 -- 172.21.15.173:0/4128309899 >> 172.21.15.173:0/4128309899 conn(0x7f23480f1aa0 msgr2=0x7f23480f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:58.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.893+0000 7f234c951700 1 -- 172.21.15.173:0/4128309899 shutdown_connections 2024-02-19T22:06:58.894 INFO:teuthology.orchestra.run.smithi173.stderr:2024-02-19T22:06:58.893+0000 7f234c951700 1 -- 172.21.15.173:0/4128309899 wait complete. 2024-02-19T22:06:58.894 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-02-19T22:06:58.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:58 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: cluster 2024-02-19T22:06:57.353837+0000 mgr.smithi143.bissbc (mgr.14162) 389 : cluster [DBG] pgmap v373: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:59.637 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-19T22:06:59.639 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-19T22:06:59.639 DEBUG:teuthology.orchestra.run.smithi143:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:06:59.664 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:06:59.683 INFO:tasks.cephadm:Stopping all daemons... 2024-02-19T22:06:59.683 INFO:tasks.cephadm.mon.smithi143:Stopping mon.smithi143... 2024-02-19T22:06:59.684 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143 2024-02-19T22:06:59.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: audit 2024-02-19T22:06:58.892848+0000 mon.smithi143 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.173:0/4128309899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:59.910 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:59 smithi143 systemd[1]: Stopping Ceph mon.smithi143 for 38187086-cf71-11ee-95bb-87774f69a715... 2024-02-19T22:07:00.411 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T22:06:59.915+0000 7f7ab271f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi143 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T22:07:00.411 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:06:59 smithi143 ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143[38075]: debug 2024-02-19T22:06:59.915+0000 7f7ab271f700 -1 mon.smithi143@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T22:07:00.847 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Feb 19 22:07:00 smithi143 podman[43954]: ceph-38187086-cf71-11ee-95bb-87774f69a715-mon.smithi143 2024-02-19T22:07:00.851 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi143.service' 2024-02-19T22:07:00.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T22:07:00.903 INFO:tasks.cephadm.mon.smithi143:Stopped mon.smithi143 2024-02-19T22:07:00.904 INFO:tasks.cephadm.mon.smithi173:Stopping mon.smithi173... 2024-02-19T22:07:00.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi173 2024-02-19T22:07:00.941 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi173.service: Unit ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi173.service not loaded. 2024-02-19T22:07:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-19T22:07:00.944 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi173" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi173 with status 5: 'sudo systemctl stop ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi173' 2024-02-19T22:07:00.945 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-19T22:07:00.947 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781/remote/ubuntu@smithi143.front.sepia.ceph.com/crash 2024-02-19T22:07:00.948 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:07:00.996 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781/remote/ubuntu@smithi173.front.sepia.ceph.com/crash 2024-02-19T22:07:00.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:07:01.041 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/38187086-cf71-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-19T22:07:01.042 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-02-19T22:07:01.043 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-19T22:07:01.043 DEBUG:teuthology.orchestra.run.smithi143:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'Degraded data redundancy' | egrep -v '\(PG_' | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'deprecated feature inline_data' | egrep -v 'compat changed unexpectedly' | egrep -v 'overall HEALTH_' | egrep -v OSD_DOWN | egrep -v OSD_ | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-19T22:07:01.078 INFO:tasks.cephadm:Compressing logs... 2024-02-19T22:07:01.078 DEBUG:teuthology.orchestra.run.smithi143:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:07:01.120 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:07:01.146 INFO:teuthology.orchestra.run.smithi143.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T22:07:01.146 INFO:teuthology.orchestra.run.smithi143.stderr:: No such file or directory 2024-02-19T22:07:01.146 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-mgr.smithi143.bissbc.log 2024-02-19T22:07:01.146 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:07:01.147 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-mgr.smithi143.bissbc.log: gzip -5 --verbose -- /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-19T22:07:01.147 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-19T22:07:01.147 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph.log 2024-02-19T22:07:01.148 INFO:teuthology.orchestra.run.smithi143.stderr: 88.4% -- replaced with /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:07:01.148 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-mon.smithi143.log 2024-02-19T22:07:01.148 INFO:teuthology.orchestra.run.smithi143.stderr: 90.3% -- replaced with /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-19T22:07:01.148 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph.log: 88.9% -- replaced with /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-19T22:07:01.148 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-mon.smithi143.log: 88.8% -- replaced with /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-19T22:07:01.151 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T22:07:01.151 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-02-19T22:07:01.151 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:07:01.152 INFO:teuthology.orchestra.run.smithi143.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:07:01.152 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:07:01.153 INFO:teuthology.orchestra.run.smithi173.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:07:01.155 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-02-19T22:07:01.155 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m0.015s 2024-02-19T22:07:01.155 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m0.011s 2024-02-19T22:07:01.155 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m0.014s 2024-02-19T22:07:01.244 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-mgr.smithi143.bissbc.log: file size changed while zipping 2024-02-19T22:07:01.244 INFO:teuthology.orchestra.run.smithi143.stderr: 92.8% -- replaced with /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-mgr.smithi143.bissbc.log.gz 2024-02-19T22:07:01.324 INFO:teuthology.orchestra.run.smithi143.stderr: 90.6% -- replaced with /var/log/ceph/38187086-cf71-11ee-95bb-87774f69a715/ceph-mon.smithi143.log.gz 2024-02-19T22:07:01.325 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-02-19T22:07:01.325 INFO:teuthology.orchestra.run.smithi143.stderr:real 0m0.190s 2024-02-19T22:07:01.325 INFO:teuthology.orchestra.run.smithi143.stderr:user 0m0.277s 2024-02-19T22:07:01.325 INFO:teuthology.orchestra.run.smithi143.stderr:sys 0m0.028s 2024-02-19T22:07:01.326 INFO:tasks.cephadm:Archiving logs... 2024-02-19T22:07:01.326 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781/remote/ubuntu@smithi143.front.sepia.ceph.com/log 2024-02-19T22:07:01.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:07:01.453 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781/remote/ubuntu@smithi173.front.sepia.ceph.com/log 2024-02-19T22:07:01.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:07:01.489 INFO:tasks.cephadm:Removing cluster... 2024-02-19T22:07:01.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38187086-cf71-11ee-95bb-87774f69a715 --force 2024-02-19T22:07:04.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38187086-cf71-11ee-95bb-87774f69a715 --force 2024-02-19T22:09:28.903 INFO:tasks.cephadm:Removing cephadm ... 2024-02-19T22:09:28.903 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:09:28.928 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:09:28.946 INFO:tasks.cephadm:Teardown complete 2024-02-19T22:09:28.947 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi173 with status 5: 'sudo systemctl stop ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi173' 2024-02-19T22:09:29.182 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=720d3b4e826f4ba0b58baa9cbd014ee7 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi173 with status 5: 'sudo systemctl stop ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi173' 2024-02-19T22:09:29.186 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-19T22:09:29.196 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-19T22:09:29.244 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-19T22:09:29.244 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:09:29.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:09:29.367 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-19T22:09:29.379 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-19T22:09:29.379 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:09:29.382 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:09:29.400 INFO:teuthology.orchestra.run.smithi143.stderr:bash: ntpq: command not found 2024-02-19T22:09:29.401 INFO:teuthology.orchestra.run.smithi173.stderr:bash: ntpq: command not found 2024-02-19T22:09:29.405 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:09:29.405 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-02-19T22:09:29.405 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +1323us[+1323us] +/- 89ms 2024-02-19T22:09:29.405 INFO:teuthology.orchestra.run.smithi143.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -1430us[-1435us] +/- 57ms 2024-02-19T22:09:29.405 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +691us[ +685us] +/- 75ms 2024-02-19T22:09:29.405 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:09:29.407 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:09:29.407 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-02-19T22:09:29.407 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 +1315us[+1319us] +/- 89ms 2024-02-19T22:09:29.407 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -1435us[-1431us] +/- 58ms 2024-02-19T22:09:29.407 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +691us[ +691us] +/- 75ms 2024-02-19T22:09:29.407 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:09:29.408 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-19T22:09:29.444 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-19T22:09:29.445 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-19T22:09:29.511 DEBUG:teuthology.orchestra.run.smithi143:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:09:29.579 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:09:29.651 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:09:29.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:09:29.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:09:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:09:29.717 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-19T22:09:29.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-19T22:09:29.778 INFO:teuthology.task.internal:Duration was 1588.819635 seconds 2024-02-19T22:09:29.779 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-19T22:09:29.842 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-19T22:09:29.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:09:29.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:09:29.892 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:09:29.897 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:09:30.190 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-19T22:09:30.190 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2024-02-19T22:09:30.190 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:09:30.214 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-02-19T22:09:30.215 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:09:30.240 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-19T22:09:30.240 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:09:30.256 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:09:30.332 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-19T22:09:30.333 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:09:30.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:09:30.432 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-19T22:09:30.442 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-19T22:09:30.442 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:09:30.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:09:30.495 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-19T22:09:30.505 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:09:30.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:09:30.535 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2024-02-19T22:09:30.562 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-02-19T22:09:30.617 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:09:30.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:09:30.634 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:09:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:09:30.675 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-19T22:09:30.686 INFO:teuthology.task.internal:Transferring archived files... 2024-02-19T22:09:30.687 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781/remote/smithi143 2024-02-19T22:09:30.687 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:09:30.727 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781/remote/smithi173 2024-02-19T22:09:30.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:09:30.766 INFO:teuthology.task.internal:Removing archive directory... 2024-02-19T22:09:30.767 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:09:30.769 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:09:30.818 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-19T22:09:30.829 INFO:teuthology.task.internal:Not uploading archives. 2024-02-19T22:09:30.829 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-19T22:09:30.877 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-19T22:09:30.877 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:09:30.879 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:09:30.895 INFO:teuthology.orchestra.run.smithi143.stdout: 525832 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:09 /home/ubuntu/cephtest 2024-02-19T22:09:30.897 INFO:teuthology.orchestra.run.smithi173.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:09 /home/ubuntu/cephtest 2024-02-19T22:09:30.898 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-19T22:09:30.960 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-19T22:09:31.127 INFO:teuthology.nuke:Checking targets against current locks 2024-02-19T22:09:31.161 DEBUG:teuthology.nuke:shortname: smithi143 2024-02-19T22:09:31.161 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:09:31.210 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:30:34.437686', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHyUUq7G/aCUWjKEixnVGNXk6NckngxCGMXn1sh+yFhF3Ccv6o5Nxpbw9rtT4QGa0y5SS7r/U4TdDejajeTuwsk='} 2024-02-19T22:09:31.211 DEBUG:teuthology.nuke:shortname: smithi173 2024-02-19T22:09:31.211 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:09:31.250 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566781', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:30:34.439147', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEutp+TVxGTaagfKrGqAAJK5kF2QB/T19msqiiot6i+abtSbFTMGkxg2zUwQ++Z/Rhk4Srl20LgtjHD9EMhrqFM='} 2024-02-19T22:09:31.298 INFO:teuthology.orchestra.console.smithi143:Power off 2024-02-19T22:09:31.298 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:09:31.347 INFO:teuthology.orchestra.console.smithi173:Power off 2024-02-19T22:09:31.347 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:09:31.355 DEBUG:teuthology.orchestra.console.smithi143:power off output: Chassis Power Control: Down/Off 2024-02-19T22:09:31.356 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:09:31.396 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-02-19T22:09:31.397 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:09:31.406 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-02-19T22:09:31.445 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-02-19T22:09:35.408 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:09:35.446 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:09:35.518 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-02-19T22:09:35.556 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-02-19T22:09:39.524 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:09:39.557 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:09:39.633 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off 2024-02-19T22:09:39.665 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-02-19T22:09:39.735 INFO:teuthology.orchestra.console.smithi143:Power off completed 2024-02-19T22:09:43.667 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:09:43.776 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-02-19T22:09:43.877 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-02-19T22:09:43.981 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1588.8196353912354 failure_reason: 'Command failed on smithi173 with status 5: ''sudo systemctl stop ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi173''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=720d3b4e826f4ba0b58baa9cbd014ee7 status: fail success: false 2024-02-19T22:09:43.982 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T22:09:44.059 INFO:teuthology.run:FAIL